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Executive Summary 



F or the past three years, the Consortium on Chicago School Research, 
in partnership with Mills College in Oakland, California, has been 
studying the implementation and impact of the Chicago High School 
Redesign Initiative (CHSRI). This partnership between the Chicago Public 
Schools (CPS), the Gates Foundation, and local Chicago foundations 
began in 2001. More than $26 million has been dedicated toward the goal 
of opening approximately two dozen small high schools across the city. 

The Need for Reform in Urban Public High Schools 

Urban public high schools, particularly those serving low-income students of color, 
frequently fail to prepare their students for college, work, or life as productive citi- 
zens. In Chicago, for example, 

• Only 54 percent of the 2000-01 freshman cohort graduated in four years.^ 

• The percentage of Chicago’s eleventh-graders who met the 2004 Prairie State 
Achievement Exam standards was much lower than in the state of Illinois (36 
percent vs. 57 percent in reading, and 28 percent vs. 53 percent in math).^ 

• Only 6.5 percent of those who started as thirteen-year-olds in Chicago’s public 
high schools in 1998 or 1999 had earned a bachelor’s degree by the time they were 
25. Only about 3 percent of male African-American and Latino students did so.^ 



Our Study 

This report details our quantitative analysis of how small schools compare to 
the rest of Chicago public schools, taking into account individual- and school- 
level characteristics. We used survey data to study the experiences of students 
and teachers in these schools and school district records to analyze a variety of 
educational outcomes. 

• Student Experiences : We examined students’ perceptions of instructional experi- 
ences, of teachers’ expectations for them, and of the nature of the academic and 
personal support they received. 
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• Teacher Experiences: We examined indicators 
detailing teachers’ perceptions of whether their 
school context might support reform, whether they 
experienced opportunities that facilitated instruc- 
tional improvement, and the nature of the quality 
of student discussion in their classrooms. 

• Student Outcomes: We examined student absences; 
whether students were on-track to graduate after 
freshman year; one-, two-, and three-year cumula- 
tive dropout rates; and scores on the Prairie State 
Achievement Exam. 

Findings 

Mixed results regarding teacher and student experiences 

• Teachers in CHSRI small schools were much more 
likely to report working in contexts characterized hy 
teacher influence, innovation, collective responsibil- 
ity, and teacher-teacher trust than similar teachers 
in other Chicago high schools. 

• CHSRI teachers were slightly more likely to report 
program coherence and opportunities for reflec- 
tive dialogue, professional development, and other 
facilitators of instructional improvement. However, 
these differences were small and not statistically 
significant. 

• Student and teacher reports of instruction in small 
schools were generally quite similar to reports in 
other schools with comparable students. One notable 
exception is that juniors in small schools reported 
greater levels of academic press; this difference was 
statistically significant. 

• We saw substantial evidence that juniors experi- 
enced moderately higher expectations and that they 
received more academic and social support than 
similar students at other public high schools in 
Chicago. To a more limited degree, this was also 
true for first-time freshmen. 

Mixed results regarding student outcomes 

• First-time freshmen attending CHSRI schools 
were absent fewer days than their peers at other Chicago 
public high schools, a difference that varied from 



nine fewer days in 2002-03 to six fewer days during 
academic year 2004-05. This difference was statisti- 
cally significant in all three years. Juniors at CHSRI 
schools were also absent fewer days than similar 
students, but this difference was only statistically sig- 
nificant in 2002-03. In that year, juniors at CHSRI 
schools were absent nine fewer days than their peers 
in other similar Chicago schools; this difference had 
shrunk to four days by 2004-05. 

• First-time freshmen at CHSRI schools were more 
likely to be on-track to graduate than similar stu- 
dents at similar schools in all three years, but the 
difference was not large enough to be statistically 
significant. The difference ranged from about 9 
percentage points for academic year 2002-03 to 
about three percentage points in 2004-05. 

• The one-year dropout rate for first-year freshmen 
at CHSRI schools was not significantly different 
from the one-year dropout rate for similar students 
at schools serving comparable students in any of the 
three years of this study. However, by junior year, 
students attending the first cohort of CHSRI schools 
were seven percentage points less likely to have 
dropped out than similar students at non-CHSRI 
schools, a difference that is marginally significant. 

• Juniors at CHSRI schools did not score differently 
on the Prairie State Achievement Exam than similar 
students at non-CHSRI schools in either reading or 
math in any of the three years of this study. 

Implications and Conclusions 

Given both the newness of the reform and the fact that 
only five schools have been operating for three years, 
it is clearly too soon to make broad claims about the 
efficacy of small school reform in Chicago. Our find- 
ings, however, do provide some reasons for cautious 
optimism about the reform and also highlight some 
issues in need of attention. 

• It appears that small schools are fostering more 
personal and supportive contexts for both teachers 
and students. These differences may be related to 
the differences in dropout rates and absences that 
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we found in our analysis, but they do not appear 
to be spurring increased instructional reform activ- 
ity, differing instructional practices, or improved 
student achievement test scores. 

• Specifically, the cumulative three-year dropout rate 
for students attending the first cohort of CHSRI 
schools is 7 percentage points lower than it is for 
similar students attending other CPS schools that 
serve comparable students (20 percent vs. 27 per- 
cent). This is a sizable difference. Furthermore, 
for the second cohort of CHSRI schools, there is 
already a two-year cumulative dropout rate that 
is 3 percentage points lower than it is for similar 
students attending other comparable CPS schools. 
From some perspectives, if such differences persist 



in future years, it would go a long way towards 
justifying small school reform. 

• Instructional reform efforts, instructional practice, 
and academic test scores all appear the same at small 
schools as at other CPS schools serving comparable 
students. This represents a sizable shortcoming of 
the reform effort. In response to this concern, dis- 
trict leaders are now coordinating their efforts with 
the CHSRI staff to help schools focus more directly 
and productively on instructional improvement. 
It will be important to follow these efforts, as we 
suspect that their success is essential if educators 
in small schools are to be able to foster improved 
academic outcomes for their students. 



Endnotes 

1 Allensworth (2005). 

2 Ponisciak (2005). 

3 Roderick, Nagaoka, and Allensworth (2006); Roderick (2006). 
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Introduction 



I ncreasingly, researchers, policymakers, school leaders, and concerned 
citizens are recognizing that high schools in the United States are in 
need of major reform — or, as some have put it, in need of being reinvented/ 
Current research shows that high schools are not preparing students for 
college, work, or life,^ and that they are leading to increased alienation 
among students.^ In a much-noted speech to the National Governors 
Association, Bill Gates described high schools as “obsolete.” He continued, 
“By obsolete, I don’t just mean that our high schools are broken, flawed, and 
under-funded. Although a case could be made for every one of those points. 
By obsolete, I mean that our high schools, even when they are working 
exactly as designed, cannot teach our kids what they need to know today.” ^ 
Analysis by Greene and Winters indicates that the national graduation 
rate for the class of 2002 was 71 percent for public school students, and that 
only 34 percent of students who entered ninth grade in public schools left 
school with both a regular diploma and the qualifications to attend a four-year 
college. The problem is especially severe in large urban high schools, which 
disproportionately serve students of low socioeconomic status and students 
of color. According to Greene and Winters, of students enrolled as ninth- 
graders and scheduled to graduate in 2002, only 52 percent of Latino and 56 
percent of African-American students ultimately earned a regular diploma. 
Furthermore, the likelihood of graduating with the abilities and qualiflcations 
to even apply to a four-year institution is only 40 percent for white students, 23 
percent for African-American students, and 20 percent for Latino students.^ 
Ghicago’s public schools reflect these trends. Only 54 percent of the 2000-01 
freshman cohort graduated in four years, by 2005.^ In addition, eleventh- 
graders in Illinois scored higher than eleventh-graders in Ghicago on the 2004 
Prairie State Achievement Exam (PSAE) (57 percent of Illinois students met 
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standards in reading compared with 36 percent in 
Chicago, and 53 percent of all students in Illinois 
met standards in math compared with 28 percent 
in Chicago). 7 Moreover, Roderick, Nagaoka, and 
Allensworth found that only 6.5 percent of 13-year-olds 
in Chicago’s public high schools in 1998 or 1999 had 
graduated from high school and earned a bachelor’s 
degree by their mid 20s (within six years) . And only 
about 3 percent of male African-American and Latino 
students did so.® As Daniels, Bizar, and Zemelman 
write, “America’s high schools are failing all of our kids 
some of the time and some of our kids all the time.”9 
Spurred by these concerns, many educational 
reformers have proposed that the creation of small 
schools provides a possible response to Powell, Farrar, 
and Cohen’s characterization of comprehensive 
“shopping mall” high schools as impersonal, incoher- 
ent, and ineffective. 10 This focus reverses a trend often 
associated with James Bryant Conant who, roughly 
50 years ago, argued that small rural schools were 
less effective than larger comprehensive high schools 
that could provide students with greater opportunities 
through an appropriately differentiated curriculum, n 
Reform focused on smaller, more personal schools 
has been spurred by educators such as Deborah Meier, 
researchers such as Fred Newmann and Gary Wehlage, 
and by foundation funding. 12 Most notably, the Bill 
and Melinda Gates Foundation has committed over 
$900 million to high school reform, most of it com- 
mitted to improving schools through the creation 
and replication of small high schools. 13 Energized by 
these efforts, the city of Chicago and numerous other 
urban districts are emphasizing the creation of small 
schools as a key part of their high school improvement 
strategies. 



Studying Small School Reform in Chicago 

For the past three years the Consortium on Chicago 
School Research, in partnership with Mills College, 
has been studying the implementation and impact of 
small school reform in Chicago. A major goal of this 
work has been to assess whether students attending 
small high schools had improved academic perfor- 
mance and lower dropout rates when compared with 
similar students attending other schools in the dis- 
trict. In particular, we wanted to know: Are students 
absent less often? Are they more likely to be on-track 
to graduate by the end of their freshman year? Are they 
less likely to drop out? Are they performing better on 
state achievement tests? 

In addition, in an effort to understand why there 
might or might not be differences in these outcomes, 
we assessed whether small schools fostered environ- 
ments for teachers and students that differed from the 
environments in larger schools serving similar students. 
We were especially interested in whether small schools 
created work environments for teachers that spurred 
instructional innovation and, ultimately, improved 
classroom opportunities for students to learn, as well 
as whether small schools promoted more academically 
and personally supportive settings for students. 

We begin by reviewing some pertinent research on 
small high schools and why they may hold promise 
for improving high school outcomes. Drawing on this 
review, we will next describe the theory of action un- 
derlying Chicago’s small school reform. We then detail 
the methods associated with our quantitative inquiry, 
the key findings, and potential implications. 



Endnotes 

1 Harvey and Housman (2004). 

2 American Diploma Project (2004) 

3 American Youth Policy Forum (2001). 

4 Gates (2005). 

5 Greene and Winters (2005). 

6 Allensworth (2005). 

7 Ponisciak (2005). 



8 Roderick, Nagaoka, and Allensworth, (2006); see also Roderick, 
(2006). 

9 Daniels, Bizar, and Zemelman (2001), 22. 

10 Powell, Farrar, and Gohen (1985). 

11 Gonant (1959). 

12 Meier (1995); Newman and Wehlage (1995). 

13 Bill and Melinda Gates Foundation (2006); Hendrie (2004). 
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Prior Research on the Impact of 
Small Schools 

Are Small Schools Associated with Improved Outcomes? 

F indings regarding the impact on students of small school reform efforts 
are not consistent. Several studies have found that small elementary 
and secondary schools are associated with improved student achievement.^ 
There is also evidence that small schools promote more equitable access to 
academically demanding courses,^ more equitable gains in achievement,^ and 
lower dropout rates. ^ In addition, Barker and Gump’s seminal study demon- 
strated that small schools tend to provide students with more opportunities 
for participation and leadership in a wide range of extracurricular activities.^ 
Not all findings regarding the impact of small schools are positive, 
however. Wasley et ah, as well as Hess and Cytrynbaum, studied small 
schools in Chicago and found enhanced engagement, but no consistent 
impact on student achievement.^ Similarly, preliminary results from the 
national evaluation of the Gates small schools initiative are mixed. While 
researchers found evidence of a more positive school climate, especially in 
terms of increased personalization for students and increased common focus 
for teachers, they also found continuing deficits in teacher capacity two or 
three years into the reform.^ In addition, as part of their 2004 evaluation, 
researchers also collected teacher assignments and student work from 16 
schools. They found some evidence of more rigorous teacher assignments 
and higher-quality student work in English/Language Arts. However, they 
also found that student work in mathematics was actually of lower quality 
in small schools than in the schools with which they were compared.^ The 
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national study also compared improvement trends in 
the test scores of small schools in three districts with 
those of the rest of the schools in those districts. The 
results of these comparisons are mixed and therefore 
inconclusive.9 

Which Characteristics Are Associated 
with Successful Small Schools? 

Just as the evidence regarding outcomes in small 
schools is equivocal, so is the evidence regarding which 
small school components are most essential or most 
highly related to improved achievement. Some scholars 
have sought to identify these factors, indicating that the 
henehcial value of small schools depends on the degree 
to which they promote factors such as personalization, 
interactive and authentic instruction, and challenging 
curriculum, while also promoting equitable student 
learning opportunities, lo 

Unfortunately, promoting these characteristics 
appears to he quite difficult. Research on small learn- 
ing communities and instructional practices finds that 
changes in teachers’ professional environment are more 
common than changes in curriculum and instruc- 
tion. For example, a study of initiatives to create small 
learning communities in Cincinnati and Philadelphia 
found that they improved the communal culture of 
teacher communities, hut did not alter the instructional 
focus. The qualitative work we have been doing in 
Chicago, which examines efforts to promote instruc- 
tional change in small schools, has come to similar con- 
clusions. Teachers report a more collaborative context, 
but do not report that this is altering their instructional 
practices in a significant way, unless there is evidence 
of strong instructional leadership. 12 

Adding to the uncertainty is the question of scale. 
Many current reform efforts, including Chicago’s, go 
far beyond launching a few idiosyncratic small schools; 



instead these initiatives aim to promote small schools 
on a large (or at least larger) scale. This more expansive 
reform strategy is worthy of attention. To date, many 
small schools have been launched in a relatively idio- 
syncratic way. Their leaders may have had particular 
insights or vision; special connections; or a great deal of 
charisma, resources, drive, etc. For example, several of 
the small high schools that existed in Chicago prior to 
the Chicago High School Redesign Initiative (CHSRI) 
raised hundreds of thousands of dollars in addition to 
the funds provided by the district. Even if these smaller 
schools outperformed other high schools serving similar 
students, it does not mean that efforts to launch small 
schools on a large scale will be as effective. 

Given the scant evidence base, how can replicators 
be sure what factors must be present for small schools 
to succeed? Indeed, although much has been written 
about small high schools, and although many in the 
foundation community and some reformers believe 
that this strategy can improve the life chances of high 
school students, the research supporting these beliefs is 
limited. Moreover, while current studies indicate that 
small schools are sometimes associated with improved 
outcomes, neither the magnitude nor the consistency 
of these effects is clear, and the factors that can lead 
small schools to produce improved outcomes are not 
well understood. 

Our study aims to deepen understanding of these 
issues. We examine one particular districtwide effort 
in the Chicago Public Schools (CPS) to promote small 
school reform on a larger scale. In addition, to the 
extent that small schools are associated with outcomes 
that differ from outcomes in larger schools serving 
similar students, we hope to understand why. We 
therefore examine whether small schools create more 
academically and personally supportive contexts and 
improved instructional opportunities for students. 



Endnotes 



1 Cotton (1996); Darling-Hammond, Ancess, and Ort (2002); 
Lee (2002); Holland (2002); Haller (1993); Howley (1989). 

2 Bryk, Lee, and Holland (1993) 

3 Lee and Smith (1995); Darling-Hammond, et al. (2002) 

4 Darling-Hammond, et al. (2002); Holland (2002); Pittman and 
Haughwout (1987). 

5 Barker and Gump (1964). 



6 Wasley, et al. (2000); Hess and Cytrynbaum (2002) 

7 Shear, et al. (2005). 

8 Mitchell, et al. (2005) 

9 Rhodes, et al. (2005). 

10 Darling-Hammond, et al. (2002); Lee (2002). 

11 Supovitz and Christman (2003). 

12 Stevens (2006). 
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Small School Reform in Chicago 

C hicago provides a valuable setting for considering the challenges and 
possibilities of small school reform. It is the third-largest district in the 
United States, serving some 427,000 students. Slightly more than 106,000 
of these students are in grades 9 through 12, attending one of 107 high 
schools, including roughly two dozen small schools and charter schools. In 
addition, overall student performance in high schools is unacceptably low. 
As noted earlier, on average, high school students in Chicago significantly 
underperform the rest of the state on the PSAE, and CPS has an overall 
graduation rate of just 54 percent. In response to these problems, the district 
has instituted a wide range of policies to improve educational opportunities 
and outcomes for students. CPS has made schools accountable for student 
performance, sanctioning those that do not meet test-score performance 
targets and rewarding those that do; created a system of selective-enrollment 
schools intended to attract and retain high- achieving students; strengthened 
the list of courses required for graduation; and created both opportuni- 
ties and requirements related to professional and curricular development.^ 
In addition to the above strategies, the creation of small high schools 
is an important component of Chicago’s school reform strategy.^ This 
approach has been adopted by other districts, and sometimes by individual 
schools, leading to small schools of different forms. For example, some 
small schools exist as autonomous units within larger school buildings, 
while others are smaller learning communities with a common administra- 
tive team. Still others are newly created, freestanding, autonomous schools 
(some of which may be charter schools). Some small schools were created 
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by breaking up poorly performing large schools, and 
some were created to serve smaller student bodies be- 
cause they have highly specialized foci or admission 
requirements. All of these forms can be found within 
Chicago’s public high schools. While Chicago has a 
history of small school creation, we will focus on one 
prominent strategy for high school reform, the Chicago 
High School Redesign Initiative (CHSRI). In partner- 
ship with CPS, the effort was launched by the Bill and 
Melinda Gates Foundation and local funders. It began 
in September 2001 with a $12 million grant from the 
Gates Foundation, matched by $6 million from foun- 
dations in Chicago, and ultimately will develop and 
support approximately 25 new small high schools. 

The goal of the first phase of this initiative was to 
convert up to five large high schools into 15 to 20 
autonomous small schools over five years. CHSRI 
published a formal request for proposals (RFP) in 2001, 
indicating that large high schools would be selected 
based on criteria including having a history of small 
school activity; being a neighborhood school with high 
need factors (poverty and low academic achievement) ; 
submitting a high-quality proposal with a governance 
plan; and demonstrated support from all stakehold- 
ers of the school community.3 Teachers and other 
interested parties were invited to apply for foundation 
funds and the right to convert a large school into a 
group of autonomous small schools. These schools 
would share the building, but not administrative staff 
or discretionary funds.4 The CHSRI board and CPS 
administrators selected three large, underperforming 
high schools, and from the individual proposals from 
these schools selected five small high schools to open 
in the fall of 2002. An additional four schools in the 
same buildings opened in 2003, and a final set of three 
schools opened in 2004. Participating teachers for 
these twelve small schools came largely (although not 
solely) from the teaching force that was already at the 
three large schools. Indeed, teachers knew that they 
needed to join one of the small schools by the time 
the school’s conversion was completed or they would 
lose their job. 

In April 2003, CHSRI received a second grant of 
almost $ 8 million from the Gates Foundation to open 
twelve new (rather than converted) small high schools 



over five years.5 So, in addition to opening conversion 
schools, starting in the fall of 2004 CHSRI adopted 
a second strategy, namely starting new schools from 
scratch. In that year, CHSRI helped fund and sup- 
port the creation of two new small schools that were 
part of the Big Picture network and also worked with 
other stakeholders in Chicago to respond to additional 
RFPs. In 2005, CHSRI opened two additional new 
(not converted) schools. In these cases, faculty were 
recruited from other schools and from the general 
hiring pool. 6 

The Initiative’s Theory of Change 

Like many current evaluative studies of educational in- 
terventions, we have adopted a theory-driven strategy.7 
Rather than focusing only on whether the intervention 
promoted improved student outcomes, this strategy 
also examines the assumptions the architects of the 
initiative made regarding why or how the reform would 
achieve improved outcomes. For instance, as we will 
detail below, proponents of small school reform often 
assume that small schools are a good way to create 
more collegial environments for teachers, which in 
turn will prompt instructional innovations. They also 
often assume that efforts to reform teaching will lead 
to better instruction and this, in turn, will promote 
student learning. 

An outcome-driven study might focus solely on 
whether students in small schools learned more than 
similar students who did not attend small schools. In 
contrast, a study that employs a “theory of change” 
approach will examine whether teachers who work 
in small schools do, in fact, experience more collegial 
settings and whether they teach differently than those 
in larger schools serving similar students. Such a study 
will also examine student learning outcomes. The hope 
is that by focusing on central assumptions regarding 
why and how a given reform promotes desired results, 
researchers gain a better understanding of why a re- 
form succeeded (or failed) . They also discover which 
assumptions might be in need of revision and whether 
modifications to the reform strategy or its implementa- 
tion might be necessary. 

Therefore, an early priority in our study of CHSRI 
was to map out the reform’s theory of change or, as we 



10 



Small High Schools on a Larger Scale 



FIGURE 1 

Theory of Action 
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> Limited Bureaucracy 



o 
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> Small Size 
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Facilitators of Instructional Improvement 
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• Quality Professional Development 

• Access to New Ideas 
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• Teacher-Teacher Trust 
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• Sense of Innovation 

• Teacher Influence 
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• Quality Student Discussion 
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Supportive Student Context 
Expectations 

• Expectations for Postsecondary 

• Schoolwide Future Orientation 

Academic and Personal Support 

• Respectful Classroom Behavior 

• Peer Support for Academic 
Achievement 

• Student Sense of Belonging 

• Classroom Personalism 

• Student-Teacher Trust 

• Teacher Support 




student Outcomes 

• Dropout Rate 

• Attendance 

• On-Track Rate 

• Test Scores 

> Graduation Rate 



• Bulleted text Is the focus of this report, measured through student and teacher surveys and administrative records. 



called it, theory of action. We conducted interviews 
with key stakeholders: funders, district leaders, reform 
staff, teachers, and principals. We also examined writ- 
ten documents associated with the reform agenda (such 
as the RFP to create small schools and district state- 
ments regarding the initiative). Finally, we looked at the 
broader literature on small schools as well as statements 
hy the Gates Foundation, which local actors drew 
upon to explain what they were hoping to accomplish 
and how they were going to do it. We then drafted a 
memo that outlined a theory of action, shared it with 
key stakeholders, and made minor revisions. Although 
this theory has undergone modest revision since then 
to reflect clarifications and revised thinking on the 
reform, it has stayed largely the same and has served 
as an organizing framework for our analysis. 

Below, we describe this theory. This framework por- 
trays the mechanisms through which various features 
of small school reform are thought to promote desired 
contexts for students and teachers. It also details how 
these contexts, in the presence of district, state, and 
federal influence, can promote both curricular change 
and desired outcomes. See Figure 1. 

The theory, as outlined in Figure 1, suggests that 



improved student outcomes result from numerous fac- 
tors both inside and outside the small school. First, and 
primarily, the theory and its advocates assume that if 
a district (Box 1) provides appropriate resources and 
policies then this will lead to the creation of small, vol- 
untary schools, in which teachers and principals experi- 
ence limited bureaucratic regulation (Box 2). This, in 
turn, will help create a desirable context for reform (Box 
6) characterized by trust, collective responsibility, and 
teacher influence, for example. Moreover, it assumes 
that creating such contexts for teachers and principals 
is fundamentally important as a means of fostering a 
setting where meaningful instructional improvement 
activities can occur (Box 7). Clearly, the creation of a 
context conducive to instructional improvement activi- 
ties is also shaped by the provision of resources and 
supports from both the CHSRI staff and the district 
(Boxes 4 and 5). Those helping to shape and implement 
this initiative must deftly balance their need to provide 
support and direction with their need to protect the 
small schools’ autonomy. The primacy of small school 
autonomy and flexibility must also be balanced with the 
need for accountability and bureaucratic structures, so 
that the reform can function effectively and responsibly 
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on a large scale. Ideally, this combination of internal 
vision and external support and direction will lead to 
meaningful and coherent instructional improvement 
activities (Box 7), which should lead to higher quality 
instruction (Box 8). 

Proponents of small school reform also believe 
that it will enable creation of school communities in 
which all students are held to high expectations and 
receive both personal and academic support. In large 
part, this hope rests on the assumption that smaller 
environments will spur more personal relationships 
between students and teachers. Commitment to the 
school community should be further enhanced by the 
fact that both students and teachers have chosen to 
be part of a school and share an interest in a common 
curricular theme (Box 3). These more personal relation- 
ships and shared sense of community, combined with 
the ability to keep track of all students, are expected 
to help teachers provide greater academic and personal 
support, while holding all students to higher expecta- 
tions (Box 9). This supportive context, both on its own 
and combined with high quality instruction, will, if 
reformers’ assumptions hold, make desirable student 
outcomes more likely (Box 10). 

Although our broad study aims to examine the 
entire theory of change described above, this report 
concentrates on those aspects of the theory of change 
that we can assess quantitatively. In particular, drawing 
on Consortium surveys and administrative data from 
CPS, this report examines those aspects of the theory 
outlined in Boxes 6, 7, 8, 9, and 10. Qualitative work 
carried out as part of this project has also examined 
Boxes 1, 2, 3, 4, and 5, but we do not detail that 
research here.^ 

Students and Teachers in the 
CHSRI Schools 

As previously stated, the three high schools where 
the conversions took place were among the lowest 
performing in the city. In comparison with students 
who attend other high schools in Chicago, CHSRI 
students’ achievement scores are lower, they are more 
likely to receive special education services, more likely 
to be old for grade, more likely to have been mobile 



TABLE 1 

Performance and Demographic Characteristics of First-Time 
Freshmen, 2004-05 





CHSRI 


All Other 
High 
Schools 


All Other 
Nonselective 
High Schools 


Incoming Eighth-Grade 
ITBS Reading* 


228 


246 


240 


Racial Composition Percentage 


African-American 


85.6 


50.9 


53.0 


Latino 


11.5 


35.9 


36.8 


White 


2.2 


9.2 


7.3 


Percentage Receiving Special 


Education Services 


25.5 


17.0 


18.2 


Percentage Old for Grade 


34.9 


23.7 


25.9 


Percentage with Two or More 


Moves in the Three Years 
Prior to High School 


20.1 


12.8 


13.7 



The national average Iowa Tests of Basic Skills (ITBS) score for an eighth- 
grader is 250, for a seventh-grader is 239, and for a sixth-grader is 227. 



in elementary school, and they are more likely to be 
students of color (see Table 1).9 

As detailed in Table 1, the CHSRI freshman class 
of 2004-05 (the most recent year for which we have 
data) came into high school with an average Iowa Tests 
of Basic Skills (ITBS) score in reading of 228. To put 
this in perspective, a sixth-grade student reading at the 
national average would score 227 at the end of the year, 
which means that the 2004-05 CHSRI freshman class 
scored roughly two years below grade level. 

CHSRI schools serve a population of students that 
is overwhelmingly minority; 85 percent of students in 
CHSRI schools are African-American. In addition, 
one-quarter of the students received special education 
services and more than one-third of the students started 
ninth grade at least one year older than students who 
progressed regularly through school; in other words, 
these students had been retained at least once in their 
elementary school career. One-hfth of these students 
changed elementary schools at least twice in the middle 
school years (most of Chicago’s elementary schools 
extend through eighth grade) . 

Although Table 1 presents student characteristics 
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for 2004-05, the first two CHSRI cohorts (2003-03 
and 2003-04) are very similar in terms of achievement 
and demographic characteristics. 

To examine the teaching force in CHSRI schools, we 
looked at self-reported background information from 
teachers who took the 2005 Consortium survey (see 
Appendix A for details on the overall survey response 
rates). CHSRI schools tend to have a larger popula- 
tion of novice teachers with less than three years of 
experience (36 percent vs. 26 percent in other high 



schools) and a smaller population of veteran teachers 
with more than 1 1 years of experience (33 percent vs. 
44 percent) . In addition, a slightly higher percentage 
of teachers at CHSRI schools entered the profession 
through alternative certification. 

In short, CHSRI schools admit some of the lowest- 
performing students in the system who bring with them 
significant challenges such as high mobility, and they 
must serve these students with teachers who are less 
experienced than those in the rest of the system. 



Endnotes 

1 Lee (2002). 

2 Although this effort received a major push in late 2001, the focus 
is not new to Chicago. In 1995, the Chicago Board of Education 
endorsed a resolution (95-0829-RS2) that affirmed the value of 
developing small schools, and five small charter schools as well as 
one district small school were in operation prior to 1999. 

3 Chicago High School Redesign Initiative Request for Proposals, 
issued December 3, 2001, Section IV. Available from authors. 

4 Ibid. 

5 Chicago Public Schools, Office of Small Schools (n.d.). 

6 The conversion schools opened with a variety of grade level 
configurations due to particularities within each converting school. Of 
the 12 conversion schools, four opened with ninth-graders only, one 
opened with ninth- and tenth-graders, one with grades 9 and 1 1 , and 
six schools opened with grades 9 through 12. All four of the new starts 
opened with freshmen only. 



7 Weiss (1995); Chen (1990); Connell, et al. (1995). 

8 Stevens (2006); Sporte, Correa, Kahne, and Easton (2003); Sporte, 
Kahne, and Correa (2004). 

9 This appears to be true of Gates-supported small schools in 
general. In the four districts where SRI International and the American 
Institutes for Research examined such data, the Gates-supported small 
schools were more likely than other schools to have students with lower 
academic achievement, students eligible for free and reduced-price 
lunch, students who were members of minority groups, students 
receiving special education services, and students of limited English 
proficiency. See Rhodes, et al. (2005). 
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Sample and Methods Used in Our Research 

I n order to address the research questions in this study, we relied on students’ 
and teachers’ survey responses and student administrative and test score 
data. For the analysis of student and teacher contexts, we studied responses 
from the Consortium’s biannual districtwide survey, administered in April 
and May of 2005.^ For the teacher survey there were 10 CHSRI schools 
representing 190 teachers and 42 non-CHSRI schools representing 3,083 
teachers. Among first-time freshmen, 865 students in 12 CHSRI schools 
responded, as did 15,064 students in 59 non-CHSRI schools. Finally, we 
also analyzed the responses from 244 juniors in 6 CHSRI schools and 9,016 
juniors in 48 non-CHSRI schools. Additional details about the analytic 
samples can be found in Appendix A. Using Rasch scaling of the survey data, 
we constructed measures of the concepts described in Boxes 6, 7, 8, and 9. 
Appendix B describes the Rasch methodology. 

For the analysis of student outcomes, we examined annual data from 
CPS administrative records including grades, attendance, and scores on the 
PSAE. This permits us to look at student outcomes during the three years the 
reform has been in place. Details on sample sizes can be found in Appendix A. 

When analyzing teacher survey data we used responses from all 
high school teachers in our sample because most teachers, both in CHSRI 
schools and in other high schools, teach more than one grade level of stu- 
dents. Our analysis of student data — both context and outcome — focused on 
first-time ninth-graders as well as on eleventh-graders. There are two main 
reasons for analyzing ninth-graders’ data. First, because CHSRI schools are 
only one to three years old, ninth grade is the only grade that all CHSRI 



CONSORTIUM ON CHICAGO SCHOOL RESEARCH AT THE UNIVERSITY OF CHICAGO 



15 





schools have in common, and we wanted to compare 
similar students while maximizing our sample size. 
Second, we know from prior studies that what hap- 
pens during freshman year is of crucial importance 
to students’ future educational trajectories.2 Indeed, 
one of our outcome indicators, on-track to graduate, 
is defined specifically for first-time freshmen. 

We also have two reasons for analyzing eleventh- 
grade student data. First, one of our key outcome 
measures, standardized test scores, is based on the 
PSAE, which is only given to eleventh-graders. In ad- 
dition, it seems plausible that some of the benefits (or 
possibly costs) of small schools, such as strong social 
bonds that result from multiple years in a small school 
community, may take several years to develop. Thus, 
despite the smaller sample size, we decided to also 
compare the experiences of eleventh-graders attending 
small schools with the experiences of similar students 
attending large high schools. 

As noted above, there are some differences between 
CHSRI students and teachers and those in the rest 
of the district. Differences in survey measures or stu- 
dent outcomes could arise from the differences in the 
populations of CHSRI teachers and students versus the 
populations of other Chicago public high schools. For 
this reason, when considering how student outcomes, 
school contexts, and instruction in CHSRI schools 
compare to those in the rest of Chicago’s high schools, 
it is necessary to be sure that our comparisons are of 
similar students and teachers. Controlling for these 
characteristics allows us to estimate differences among 
students who are similar in terms of demographic and 
academic factors such as gender, race, prior achieve- 
ment, mobility, economic status, and age relative to 
grade. Similarly, we control for the impact of teacher 
outcomes that might result from differences that exist 
among teachers in terms of gender, race, education 
level, teaching experience, and certification. 

When making comparisons between CHSRI and 
the rest of the system, it is also important to consider 
how the concentration of particular student charac- 
teristics in a school could affect the differences be- 
tween CHSRI schools and other schools, beyond the 
effect of individual student attributes. Therefore, in 
our comparisons we introduce statistical controls for 



concentration effects of the academic performance 
of the students and the demographic and socio- 
economic mix of the student body in each high school. 
Finally, to account for the fact that students (or teach- 
ers) are nested within schools, we used hierarchical 
modeling.3 



How We Make Comparisons 

In the remainder of this report we will be looking 
at differences in measures and outcomes between 
students and teachers in CHSRI schools and those 
in other Chicago public schools. These compari- 
sons take individual and concentration effects into 
account, as described above. Therefore we will be 
comparing similar students and teachers under 
similar circumstances. We will not repeat this fact 
in the following sections, but the reader should 
keep in mind that all comparisons are between 
similar students or teachers in schools serving 
similar populations of students. 




Our models differed slightly depending on the 
outcome analyzed. The survey analyses used three- 
level hierarchical linear models, where the first level 
provided a latent measure score after adjusting for 
survey measurement error. At the individual teacher 
level, we controlled for race, gender, education level, 
years of experience, and whether teachers entered the 
profession through alternative certification. At the 
individual student level, we controlled for prior aca- 
demic achievement, race, gender, socioeconomic status, 
special education status, age relative to grade, and 
mobility prior to entering high school. At the school 
level we included whether the school was a CHSRI 
school as well as indicators of the aggregate incoming 
achievement level of the student body, the aggregate 
socioeconomic status of the student body, and the racial 
composition of the school. 

The analyses of student outcomes used two-level 
hierarchical models, employing similar controls at 
the individual student level as well as a set of dummy 
variables indicating the academic year. At the school 
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level, for each year we controlled for whether the school 
was a CHSRI school, the aggregate incoming achieve- 
ment level of the student body, and the aggregate 
socioeconomic status of the student body. Appendix 
C provides a complete description of the variables and 
the models we used. 

When describing differences that might exist be- 
tween CHSRI and non-CHSRI schools on the survey 
measures of school context and instruction, we state 
whether the difference is statistically signihcant and 
provide the estimated effect size of the difference. To 
say that a difference is statistically signihcant means 
that we are at least 95 percent conhdent that there 
is a difference between CHSRI and non-CHSRI 
schools on a given indicator. The effect size tells us 
how large the difference is. 4 Whether an effect size 



is “large” or important or meaningful often differs 
depending on the phenomenon being studied. 5 
A general rule of thumb is that effect sizes between 
0.20 and 0.50 are “slight,” those between 0.50 and 
0.80 are “moderate,” while those above 0.80 are “sub- 
stantial,” although in educational reform efforts the 
effects may tend to be slight to moderate but important 
nonetheless. 6 

The way we describe varying student outcomes is 
somewhat different. While we do state whether the dif- 
ference is statistically significant, we do not report effect 
sizes. Instead, we provide an estimate of the magnitude 
of the difference between students at CHSRI schools 
and similar students at other schools on the given out- 
come (dropout rates, achievement test score, etc.) . 



Interpreting Effect Sizes 

Statisticians calculate effect sizes, measured in stan- 
dard deviation units, as away of describing the impact 
of a given characteristic of a school or classroom or 
student on an outcome of interest. For example, if a 
given characteristic of a school, such as being a CHSRI 
small school, has an impact on a given outcome, such 
as Program Coherence, then we can describe the size 
of that impact in terms of its effect size. 

For those not familiar with this convention, it 
might be helpful to think of effect sizes in terms of 
where it puts one group of schools — for example, 
CHSRI schools — in relation to another group of 
schools — in this example, non-CHSRI schools. 
The size of the effect will determine how far apart 
the averages of the two groups are. An effect size of 
zero means there is no difference between the aver- 
age CHSRI and the average non-CHSRI school on 

^ ^ i 



the outcome of interest (in this example. Program 
Coherence) . 

To help understand the magnitude of these effect 
sizes, suppose we found find that the CHSRI effect 
on Program Coherence was 0.25. This would mean 
that the average CHSRI school would have a score 
that would place it above 60 percent of non-CHSRI 
schools. If the size of the effect were 0.50, then the 
average CHSRI school would have a score on that 
measure that would place it above 69 percent of 
non-CHSRI schools. A CHSRI effect size of 1.0 
for Program Coherence would mean that the average 
CHSRI school would score higher than 84 percent 
of non-CHSRI schools. Finally, if the CHSRI ef- 
fect size were 2.0, then the average CHSRI school 
would have a score that would place it above about 
98 percent of non-CHSRI schools. 



Endnotes 

1 We had usable survey data from three CHSRI schools in 2003. 
Results from that very preliminary analysis can be found in Kahne and 
Sporte (2004). 

2 See for example Allensworth and Easton (2005); Kemple, 

Herlihy and Smith (2005); Miller, Allensworth, and Kochanek (2002). 

3 Raudenbush and Bryk (2002). 



4 We measured the effect sizes in standard deviation units, and we 
computed them by dividing the estimate of the CHSRI coefficient 
by the standard deviation of the measure, determined by taking the 
square root of the sum of the level two and level three variances from 
the unconditional model for each measure. 

5 Thompson (2002). 

6 Cohen (1988); Konstantopoulos and Hedges (2005). 
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Findings 



W e organize our discussion of findings around the Theory of Action 
(See Figure 1 on page 11). For those findings related to survey measures, 
a summary of the measures, their reliabilities, and between-school variability 



can be found in Appendix D. 
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Is the context in which CHSRI teachers 
work more conducive to reform than at 
other Chicago public schools? 

Overall, we found that teachers in CHSRI schools 
reported a more supportive context for reform than 
teachers at other Chicago public schools. Specifically, 
once we controlled for the academic and demographic 
qualities of students, teachers, and their schools, we 
found that CHSRI teachers had significantly higher 
scores on all of the indicators of Box 6, Teacher Context 
for Reform, than similar teachers at other schools. 
Specifically, teachers at CHSRI schools reported 
slightly higher levels of Teacher-Teacher Trust thiLn simi- 
lar teachers at other schools (effect size = 0.37). High 
scores on this measure indicate that teachers trust other 
teachers, that they care about each other, and that they 
feel it is appropriate to discuss worries and frustrations 
with other teachers. Furthermore, they accord respect 
to other teachers who are considered to be leaders in 
school improvement or experts in their craft. 

CHSRI teachers also reported a higher sense of 
Collective Responsibility than similar teachers at other 
schools (effect size = 0.47). This measure asks teach- 
ers to estimate how many of their colleagues take 
responsibility for improving their school and for setting 
high standards for themselves. In addition, teachers 
are asked to estimate others’ sense of responsibility 
for all students in the school — for helping them learn, 
helping them develop self-control, and maintaining 
discipline. 

In addition, CHSRI teachers also indicated a greater 
level of Commitment to Innovation and engagement 
with learning than teachers at similar school (effect size 
= 0.52). This measure contains items asking teachers to 
reflect on whether teachers in the school are continually 
learning, that they have a “can do” attitude, and that 
all teachers are encouraged to stretch and grow. 

Finally, teachers at CHSRI schools reported greater 
levels of Teacher Influence than similar teachers at 
other schools, a finding that is consonant with one 
of the founding principles of the initiative — that it 
should be a teacher-led reform. The difference between 
CHSRI and non-CHSRI teachers on this measure was 
a “substantial” difference, with an effect size of 0.83. 



0 

Teacher Context for Reform 

• Teacher-Teacher Trust 

• Collective Responsibility for Student Achievement 
and School Improvement 

• Commitment to Innovation 

• Teacher Influence on Instruction, Discretionary 
Spending, In-service Content, and Other School 
Decisions 

FIGURE 2 

CHSRI Effect Size and Significance Level 
Box 6: Teacher Context for Reform 
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Items ask teachers to rate the amount of influence they 
have in decisions about instructional and curricular 
materials and programs, in setting standards for stu- 
dent behavior, in forming the content of in-service 
programs, and in administrative matters such as hiring 
new personnel and planning how to spend discretion- 
ary school funds. 

Figure 2 summarizes the results for Box 6, Teacher 
Context for Reform. 
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Do CHSRI teachers and principals 
engage more heavily in practices that 
facilitate instructional improvement 
than staff at other Chicago public schools? 

As reported above, CHSRI teachers were more likely 
to report having trusting and collegial relationships, 
a commitment to innovation, a sense of collective re- 
sponsibility for school improvement, and a belief that 
they had a stronger influence than teachers at other 
schools. That is, we observe a very favorable context 
for reform. When we look at a set of more specific 
practices thought to facilitate instructional improve- 
ment, however, CHSRI teachers reported that they 
(and their principals ) were only slightly more engaged 
with these practices than other teachers. Although all 
differences between CHSRI teachers and other teach- 
ers on measures related to instructional improvement 
activities were positive, only one was even marginally 
significant. 

The level of Principal Instructional Leadership re- 
ported by CHSRI teachers was no different from the 
level reported by teachers at other schools (effect size 
= 0.09). This measure contains items asking teachers 
whether their principal has communicated expectations 
for meeting instructional goals and has a clear vision for 
the school. In addition, teachers are asked whether their 
principal understands how children learn, sets high 
standards for teaching and learning, presses teachers 
to implement what they have learned in professional 
development, tracks student academic progress, and 
knows what is going on in classrooms. 

Furthermore, CHSRI teachers’ reports about the 
quality and quantity of professional development 
they received were not statistically different from the 
responses of other similar teachers. High scores on 
Quality Professional Development 'm6\c2XQ that teachers’ 
professional development experiences are coherent and 
sustained, include enough time to try and evaluate new 
ideas, are connected to the school’s improvement plan, 
and include opportunities to work with colleagues. 
CHSRI teachers’ responses to this measure were similar 
to the responses of other teachers (effect size = 0.13). 
In addition, there was not a statistically significant dif- 
ference between CHSRI teachers and similar teachers 
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Facilitators of Instructional Improvement 

• Principal Instructional Leadership 

• Quality Professional Development 
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Professional Development 

• Reflective Dialogue 

• Program Coherenoe 



FIGURE 3 

CHSRI Effect Size and Significance Level 

Box 7: Facilitators of Instructional Improvement - Teachers 
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on Access to New Ideas through Professional Development 
(effect size = 0.17). This measure asks teachers how 
often they participate in a network of teachers outside 
of their own school; how often they attend professional 
development activities sponsored by their school, CPS, 
or the teachers union; whether they have taken college 
courses related to improving their school; and how 
often they discuss curriculum and instruction with an 
outside organization. 

The largest CHSRI effects in Box 7, Facilitators 
of Instructional Improvement, occurred on the mea- 
sures that were most closely tied to teachers’ in-school 
practices — our measures of teacher participation in 
Reflective Dialogue and of Program Coherence. CHSRI 
teachers’ scores on Reflective Dialogue were not statis- 
tically different from the responses of other teachers 
(p = 0.15), but the effect size was 0.23, indicating 
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a slight effect. This measure asks whether teachers 
regularly discuss assumptions about teaching and 
learning and whether they share and discuss student 
work. It also asks teachers how often they discuss what 
helps students learn, school goals, developing new 
curriculum, and managing classroom behavior. CHSRI 
schools have instituted several practices that aim to 
facilitate reflection. For example, their teaching teams 
have supports for instructional improvement activi- 
ties such as schedules that enable common planning 
time and small groups of teachers working together on 
action research. Although the difference in Reflective 
Dialogue did not reach statistical significance, these 
efforts may be facilitating teachers’ reflective practices 
at CHSRI schools. 



Finally, we found marginally significant differ- 
ences (p = 0.08) in the reports of CHSRI teachers and 
teachers between other Chicago public schools when it 
came to Program Coherence. This measure asks teachers 
whether there is continuity between programs at the 
school; whether new programs are followed up; and 
whether there is coordination in curriculum, materials, 
and instruction both within and across grade levels. 
CHSRI teachers scored higher on this measure than 
similar teachers at other schools, with an effect size 
of 0.25. A substantial body of evidence suggests that 
Program Coherence is an important factor in improving 
student performance.! 

Figure 3 provides details on the size and significance 
of the CHSRI effect. 
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Do students at CHSRI schools experience 
classroom instruction differently than 
similar students at other Chicago public 
schools? 

As explained in our methods section, we examined 
the instructional experience of first-time freshmen and 
juniors separately. In general, there were few statisti- 
cally significant differences between the instructional 
experience of CHSRI students and the experiences 
of similar students attending other schools. On one 
of the three student measures there was a marginally 
significant difference between CHSRI freshmen and 
other freshmen; on another of these measures there 
was a significant difference for juniors. 

The quality of student discussion as reported by 
teachers, for example, was no higher in CHSRI schools 
than in other schools.2 CHSRI teachers’ reports of the 
degree to which students build off each others’ ideas, 
show respect for each other, and use data and text refer- 
ences to support their ideas were no different from the 
reports of teachers at other similar schools. 

Furthermore, first-time freshmen at CHSRI schools 
responded no differently than first-time freshmen at 
other schools to items asking about the quality of 
English instruction and the level of Academic Press 
they experienced (effect sizes of 0.01 and 0.04). Quality 
English Instruction includes items asking students about 
the amount of writing, revision, and attention to ele- 
ments of literature they experience, while Academic 
Press includes items asking students whether teachers 
expect everyone to work hard, and whether they find 
the work hard and are challenged. In both of these cases 
the responses of first-time freshmen at CHSRI schools 
were indistinguishable from the responses of similar 
students at other Chicago public schools. We did find 
a marginally significant difference between CHSRI 
freshmen and similar freshmen on items asking them to 
reflect on their experiences of mathematics instruction 
(effect size = 0.19). This measure asks students whether 
they discuss math problems and explain solutions to 
their classmates, apply math to real-life situations, and 
have to explain solutions in writing. 
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CHSRI Effect Size and Significance Level 
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Responses of juniors at CHSRI schools were no dif- 
ferent from the responses of their peers at other schools 
in their levels of Quality English Instruction and Quality 
Mathematics Instruction. Juniors at CHSRI schools 
did, however, report significantly more Academic Press 
than juniors at non-CHSRI schools, with an effect 
size of 0.24. 

Figure 4 provides CHSRI effect sizes and statistical 
significance levels for the one teacher measure used to 
typify instruction and for the three student measures 
for both first-time freshmen and for juniors. 
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Do CHSRI students experience a more 
supportive context than similar students at 
other Chicago high schools? 

Juniors at CHSRI schools reported that their teach- 
ers had higher expectations for them and that their 
teachers and peers provided more academic and social 
supports than similar students at other public high 
schools in Chicago. The picture is more mixed for first- 
time freshmen at CHSRI schools. Compared to other 
students, they reported that their teachers had higher 
expectations for them. Academic and social supports 
from peers and teachers are inconsistent. 

Specifically, our analysis indicates that teachers at 
CHSRI schools reported higher levels of postsecond- 
ary expectations, a measure that encompasses both the 
expectation that students will continue their educa- 
tion after high school and the willingness to help 
students plan and prepare for this step. The CHSRI 
effect for this measure was a moderately sizable 0.60. 
Furthermore, both juniors and first-time freshmen 
indicated that they felt higher levels of Schoolwide 
Future Orientation than their peers in other schools, 
with an effect size of 0.30 for freshmen and 0.34 for 
juniors. This measure asks students about the degree 
to which teachers work hard to make sure that all 
students are learning, staying in school, and planning 
for their futures, and the degree to which all students 
are encouraged to go to college. In all three cases re- 
lated to expectations — whether reported by teachers, 
freshmen, or juniors — the difference was statistically 
significant. 

The analyses of first-time freshmen indicated that 
students at CHSRI schools did not report a higher level 
of Peer Support for Academic Achievement, nor a stron- 
ger Sense of Belonging, nor more Respectful Classroom 
Behavior than their peers at other schools. The measure 
of peer support asks students about the extent to which 
they talk about what they did in class, if they help each 
other with homework, and if their friends think it is 
important to attend class. Sense of Belonging asks stu- 
dents to report on how personally connected they feel 
to the school — whether they fit in, whether there are 
people there who care about them and who will help 
them, whether the school feels like family, and whether 
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they participate in school activities. Respecful Classroom 
Behavior asks students about the norms of classroom 
behavior — how students treat each other, how often 
they disrupt class, and whether they help each other 
learn. In other words, first-time freshmen at CHSRI 
schools did not report a climate that is different from 
what is reported by students at other Chicago public 
high schools with respect to student-student interac- 
tions inside and outside of the classroom. 

On the other hand, measures that asked freshmen to 
comment on relationships between students and teachers 
indicated that these relationships are stronger in CHSRI 
schools than they are in other schools in Chicago. While 
there was no statistical difference in Student-Teacher 
Trust, there was a slight difference between first-time 
freshmen at CHSRI schools and similar first-time fresh- 
men at other schools in measures of teacher-provided 
personal and academic support. One of the measures. 
Teacher Support, asks students whether there is at least 
one teacher who cares about how they are doing, would 
be willing to help with a personal problem, and who 
would talk with them if they were having problems in 
class. Freshmen at CHSRI schools were higher than their 
peers on this measure (effect size = 0.20, marginally sig- 
nificant). The other measure. Classroom Personalism, asks 
students whether their English (or math) teacher helps 
them catch up and notices if they are having trouble 
learning. Freshmen at CHSRI schools had statistically 
stronger responses to these items than similar freshmen 
at other schools (effect size = 0.21). 



24 



Small High Schools on a Larger Scale 



CHSRI juniors, on the other hand, consistently 
scored higher than similar juniors. These differences 
were statistically significant in all but one of the six 
measures of academic and personal support. Effect 
sizes for two of the student-to-student relationships. 
Peer Support for Academic Achievement and Sense of 
Belonging, ranged from 0.34 to 0.37. The other mea- 
sure relating to student-student relationships. Respectful 
Classroom Behavior, was somewhat anomalous. While 
there was a slight CHSRI effect (0.25), the difference 
was not statistically significant. 



Indicators of student-teacher relationships for 
juniors {Classroom Personalism, Student-Teacher Trust, 
and Teacher Support) all had effect sizes ranging from 
0.32 to 0.51. Classroom Personalism and Teacher Support 
are described briefly above. Student-Teacher Trust has 
items that ask students whether teachers care about 
them, keep their promises, try to be fair, listen to stu- 
dents’ ideas, and treat students with respect. 

Figure 5 summarizes these findings for teachers and 
first-time freshmen; Figure 6 (see p. 26) summarizes 
the results for juniors. 



FIGURE 5 

CHSRI Effect Size and Significance Level 

Box 9: Supportive Student Context - Teachers and First-Time Freshmen 
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FIGURE 6 

CHSRI Effect Size and Significance Level 
Box 9: Supportive Student Context - Juniors 
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Do students at CHSRI schools exhibit 
more positive educational outcomes than 
similar students at other Chicago public 
high schools? 

We compared the outcomes of students at CHSRI 
schools with those of similar students at other Chicago 
public high schools. We assessed the impact of at- 
tending a small school on various student outcomes: 
student absences, being on-track to graduate after their 
freshman year, dropout rates, and achievement test 
results.3 Readers should keep in mind that outcome 
comparisons in year one were based on the outcomes 
of five small high schools. Six more small high schools 
were added in the initiative’s second year, so compari- 
sons of results in year two are based on results from 1 1 
small high schools. Five more were added during the 
reform’s third year (for a total of 16 schools). 

Absences 



Student Outcomes 

• Absences 

• On-Track to Graduate Rate 

• Dropout Rate 

• Achievement Test Scores 

• Graduation Rate 

FIGURE 7 

Average Number of Days Absent During the Year 
First-Time Freshmen 

CHSRI Students and Similar Students at Other Schools 

30 n 




First-time freshmen in CHSRI schools were absent less 
ofien than similar students in other schools for each of 
the three years in our study (See Figure 7). In all three 
years, these differences were statistically significant 
(p < 0.01). For example, in 2004-05 CHSRI students 
were absent 6 fewer days than similar students at other 
schools (20.1 days vs. 26.1 days). While the absence 
rates of CHSRI students appear better than those of 
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5- ♦ Other Schools 
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2004-05 



Note: P-values for significant differences: in year 2002-03, p = 0.00; 
in year 2003-04, p = 0.02; in year 2004-05: p = 0.02. 
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Definitions of Student Outcomes 

Absences: Absences are measured as the average 
number of days students were absent from their class- 
es during one academic year. Those who dropped 
out of school during the year are not included in 
the sample. 

On-Track to Graduate Rate: The on-track to graduate 
rate measures the proportion of students who have 
accumulated five credits and have no more than one 
failing grade in a semester course in a core subject 
(English, math, science, and social science) by the 
end of the freshman year. 

Dropout Rate: We identify first-time freshmen and 
follow them over time. Any student with a dropout 



code in CPS records is counted as a dropout. This 
number is divided by the initial number of first-time 
freshman less the number of students who transfer 
out of CPS. 

Achievement Test Scores: Achievement is measured 
by the scores of eleventh-graders on the Prairie State 
Achievement Exam in mathematics and English. 
This test is given to almost all high school juniors 
during the spring of their junior year and incorpo- 
rates the ACT as part of their composite score. 

Graduation Rato: Since the reform has been in place 
for less than four years, it is impossible to analyze 
four-year graduation rates at this point. 
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their counterparts, on average CHSRI students missed 
a lot of school — just over four full weeks of classes 
during the year. 

The absence rate of juniors in CHSRI schools also 
appeared better than for similar students in other 
schools (see Figure 8). However, these differences 
were only statistically significant during the 2002-03 
school year. 

On-Track to Graduate Rate 

At the end of freshman year, CHSRI students were a 
little more likely to be on-track to graduate than students 
attending comparable schools, though the differences were 
not statistically significant^ In addition, it is notable that 
the difference in being on-track to graduate appears 
to have narrowed over the past three years (see Figure 
9) and was relatively small (57 percent for CHSRI 
students and 54 percent for other students) during the 
2004-05 school year. 

Dropout Rates 

There was not a statistically significant difference in 
dropout rates between first-time freshmen at CHSRI 
schools and similar students at other schools. However, in 
subsequent years it appears that meaningful differences in 



cumulative dropout rates emerge. 

We studied the cumulative dropout rates by follow- 
ing cohorts of students starting in their freshman year 
(see Figure 10). CHSRI students in the 2002-03 cohort 
had a lower dropout rate in their first and second year 
than similar non-CHSRI students, but that difference 
was not very large. However, in their third year, the 
cumulative dropout rate for CHSRI students was 20 
percent, compared with 27 percent for similar students 
enrolled in other Chicago high schools. This difference 
was marginally significant (p = 0.056). 

Similarly, the difference in the dropout rate at the 
end of ninth grade for the 2003-04 cohort was small 
and statistically insignificant, but by the end of tenth 
grade the cumulative dropout rate was 14 percent for 
CHSRI students and 17 percent for comparable stu- 
dents attending other schools. This difference was also 
marginally significant (p = 0.08). Data regarding the 
2004-05 cohort were only available through the end 
of their freshman year. As was the case for the other 
two cohorts, no statistically significant CHSRI effect 
was found at the end of the ninth-grade year. 

This pattern mirrors data from surveys regarding 
the degree to which students feel supported and that 
they belong. Juniors attending CHSRI schools reported 



FIGURE 8 

Average Number of Days Absent During the Year 
Juniors 

CHSRI Students and Similar Students at Other Schools 



FIGURE 9 

Dn-Track to Graduate 
First-Time Freshmen 

CHSRI Students and Similar Students at Other Schools 
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much more supportive contexts than similar students 
attending other schools, hut comparisons of first-time 
freshmen yielded less sizable and fewer statistically 
significant differences. Thus, it appears that the relative 
ability of small schools to provide a supportive context 
and to hold students in school may become greater the 
longer the students remain in these settings. 

PSAE Scores 

Achievement among eleventh-graders, as measured on the 
statewide standardized achievement test, the PSAE, was 
no dijferent for students attending CHSRI high schools 
than it was for similar students at other schools. (See 
Figures 11 and 12.) We analyzed the test scores for both 
reading and math portions of the PSAE, and we did not 
find any differences in performance between students 



at CHSRI schools and similar students in other schools 
on either test. In the 2002-03 academic year, three of 
the CHSRI schools offered eleventh grade, so students 
who took the PSAE test in that year had the opportu- 
nity of being at the CHSRI school for one year. By the 
2004-05 academic year, while nine CHSRI schools had 
eleventh-graders, some of them had been in existence 
for three years, some for two, and some for one. So the 
length of time students could have attended a CHSRI 
school at the time of the PSAE differed depending 
on which school they attended. Given these added 
complexities, we tried different ways to incorporate 
these factors into our analysis. The conclusions of all 
the analyses are the same; there are no differences in 
terms of eleventh-grade achievement between CHSRI 
students and students at other CPS schools. 



FIGURE 10 

Dropout Rates by Cohort 

CHSRI Students and Similar Students at Other Schools 




Note: Significant difference in year 2004-05 only. For 2002-03 cohort, 
p = 0.056. For 2003-04 cohort, p = 0.076. 



FIGURE 11 

Average PSAE Reading Scores 
Juniors 

CHSRI Students and Similar Students at Other Schools 
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FIGURE 12 

Average PSAE Math Scores 
Juniors 

CHSRI Students and Similar Students at Other Schools 
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Trends as Schools Mature and New Schools Are Opened 



In the literature on small schools and in literature 
on reforms more generally, there is sometimes dis- 
cussion of a phenomenon known as a “sophomore 
slump.”5 The concept refers to the observation that 
schools or other organizations undergoing major 
reforms suffer setbacks during their second year of 
operation. In the evaluation of the Gates-funded 
small high schools, researchers at SRI International 
and the American Institutes for Research found that 
most schools experienced declines during the second 
year of implementation. 6 They hypothesized that 
changes in the social dynamics of the schools and 
the need to incorporate new adults who may not have 
been part of the original vision could contribute to 
this occurence, as could changes in district policies, 
leadership turnover, and teacher capacity.7 Such a 
pattern may have emerged in the small high schools 
being launched in Chicago. 

Consider, for example. Table 2, which examines 
outcomes for first-time freshmen broken down by 
the year when the school was launched. Schools that 
began in 2002-03 and in 2003-04 both experienced 



setbacks in outcomes during their respective second 
years of operation. By their third year, schools that 
started in 2002-03 appeared to bounce back some- 
what on measures of their one-year dropout rate 
and of the number of students who were on-track to 
graduate. Clearly, our ability to determine whether a 
“sophomore slump” exists will improve in the next 
year or two when we have data on additional cohorts 
of new starts. 

In addition, it is worth remembering that at every 
point in time in this study there are CHSRI schools 
in different stages of development. For example, dur- 
ing the 2003-04 academic year there were two waves 
of CHSRI schools: the five CHSRI schools that had 
been operating for a year and the six schools that 
had just opened. Therefore interpreting the results 
in 2003-04 is more challenging than interpreting 
the results in 2002-03, when there was only one 
wave of schools. It 2004-05 there were three waves 
of schools, each wave at a different point in its 
developmental trajectory. 



TABLE 2 

Freshman Outcomes for Different Groups of CHSRI Schools 







2002-03 


2003-04 


2004-05 


Number of Days Absent 


2002-03 Start-ups 


15.3 


20.6 


21.0 




2003-04 Start-ups 




19.9 


24.2 




2004-05 Start-ups 






15.6 




ALL SCHOOLS 


15.3 


20.2 


20.1 


On-Track to Graduate Rate 


2002-03 Start-ups 


62.6 


49.9 


58.6 


(percentage) 


2003-04 Start-ups 




57.3 


47.2 




2004-05 Start-ups 






63.9 




ALL SCHOOLS 


62.6 


53.2 


56.9 


One-Year Dropout Rate 


2002-03 Start-ups 


7.2 


8.8 


7.4 


(percentage) 


2003-04 Start-ups 




4.8 


6.5 




2004-05 Start-ups 






3.9 




ALL SCHOOLS 


7.2 


6.9 


5.9 



30 



Small High Schools on a Larger Scale 




Endnotes 

1 Newmann et al. (2001). 

2 The survey asked teachers about a number of their instructional 
practices, including the nature and frequency of homework they 
assign, classroom teaching strategies, amount of individualized 
instruction teachers offer, and the quality of student discussion in 
their classrooms. In all of the measures there was more variation 
between teachers within schools than there was between schools. 

The only measure where more than 5 percent of the variance was be- 
tween schools was the measure on student discussion. Since this study 
is mainly concerned with between-school differences, this is the only 
measure we report here. 

3 Recall that we estimate differences using a two-level hierarchical 
linear model and control for student characteristics and school effects. 

4 On-track rates are only computed for first-time freshmen, so we 
have no comparable finding for juniors. 

5 Borman (2005); Shear, et al. (2005). 

6 Mitchell, et al (2005). 

7 Shear, et al. (2005), pp. 27-28. 
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Discussion and Implications 

W hen discussing the findings described above, caution is clearly in order. 

We have only been able to study small school reform in Chicago for 
three years. Many of the schools we are studying are just getting started — only 
five schools have been open for three years — so we can only provide an early 
assessment of the potential of this small school initiative. Indeed, a reform as 
complex as CHSRI will require time to fully take shape. We will be able to speak 
with greater confidence regarding the effects of this initiative as the sample 
of schools and students who have fully experienced small high schools grows. 

At the same time, given the importance of small school reform and of high 
school reform more generally, we think it makes sense to take a systematic look 
at what we have found so far. Policymakers, funders, and educational leaders are 
understandably interested in emerging findings and trends. These early find- 
ings can help those implementing small high schools identify issues and ques- 
tions that deserve greater attention as the reform (and the studies of it) develop. 

Mixed Results 

Our analysis of results contains some good news as well as some reasons for concern. 
With respect to outcomes, perhaps the clearest finding is that the attendance rates 
of first-time freshmen at small high schools are better than for similar students at 
other Chicago public high schools. Over the course of the first three years, fresh- 
men attending CHSRI small high schools were absent between six and nine fewer 
days each year than similar students at schools with comparable student popula- 
tions. Given how often CPS students are absent and the importance of ninth-grade 
performance, this appears to be a very desirable outcome. 

In addition, the lower cumulative dropout rate by the junior year for students 
in the first cohort compared to similar students at high schools with comparable 
student populations (20 percent vs. 27 percent) is quite promising. Furthermore, 
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students in the second cohort of CHSRI schools 
appear to he following the same pattern, with a 
marginally significant difference between students 
in CHSRI schools compared to similar students at 
non-CHSRI schools (14 percent vs. 17 percent). This 
clearly signals a tangible benefit of small high schools. 
If such differences persist in future years, it would go 
a long way toward justifying small school reform. 

Not all the news is good, however. Eleventh-grad- 
ers’ achievement test scores are the same for students 
attending CHSRI schools as for similar students at 
non-CHSRI schools. This signals a sizable shortcoming 
of the initiative. A prime rationale for small schools was 
that the reform would spur instructional innovation 
and, ultimately, improve academic outcomes. Neither 
our survey results from spring 2005 nor our qualitative 
work in 2004-05 indicate that this is happening. 

Assessing Our Theory of Action 

Broadly speaking, the theory posits two paths to im- 
proved outcomes — one through improved instruction 
and the other through higher expectations and more 
personalized academic and social supports. Our find- 
ings can help us assess both of these paths. i 

Assumptions Regarding Expectations and Personaiism 

Small schools appear to promote higher expectations 
and greater personal and academic support for students. 
As the theory of action anticipated, such a context was 
associated with lower dropout rates. Indeed, eleventh- 
graders attending CHSRI schools reported a more sup- 
portive context than similar students in other schools 
(Box 9), and the dropout rate of eleventh-graders 
who started high school at CHSRI schools was lower 
than those who started elsewhere. Among first-time 
ninth-graders, differences in the levels of academic 
and social support between CHSRI and other schools 
were smaller and less often statistically significant. 
Consistent with this finding, there were much smaller 
and statistically insignificant differences in the one-year 
dropout rates between ninth-grade students attending 
CHSRI schools and similar students attending other- 
wise similar high schools. 

Assessing the theory’s assumption that a more 
supportive context for students could lead to better 



attendance is somewhat more complicated. CHSRI 
freshmen were not significantly different from similar 
freshmen at other schools on measures related to peer 
support, nor were they different on measures of stu- 
dent-teacher trust. They did report stronger teacher 
support, and perhaps it is this dimension that can help 
explain why they were absent significantly fewer days. 
Conversely, CHSRI juniors were different from non- 
CHSRI juniors on both peer-to-peer indicators and 
on student-teacher indicators. Yet, after the 2002-03 
school year, there were not statistically significant dif- 
ferences in the absences of juniors at CHSRI schools 
and juniors at other schools. 

Assumptions Regarding instructionai improvement 

The CHSRI small schools did not appear to promote 
instructional improvement activity, instructional qual- 
ity, or improved academic outcomes for students. The 
theory assumed that small schools would foster an im- 
proved context for reform among teachers (Box 6) and 
that this combined with supports from CHSRI staff 
(Box 4) and the district (Box 5) would spur instruc- 
tional improvement activity (Box 7). These improve- 
ment efforts, in turn, would strengthen the quality of 
instruction, which would improve test scores. Although 
working in a small school did appear to encourage a 
greater sense of teacher collegiality, trust, collective re- 
sponsibility, and related contextual features that might 
enable instructional reform activity (see Box 6), we did 
not see evidence that these contexts fostered more of 
the practices thought to facilitate instructional reform 
(Box 7). We also saw evidence that instruction was the 
same for similar students in CHSRI and non-CHSRI 
settings (Box 8). Given that we did not see a CHSRI 
effect on instruction, it is not surprising that we failed 
to find evidence that attending a small school promoted 
higher test scores. 

While test scores might be directly linked to in- 
struction, the theory also posited that improved and 
more engaging instruction might lower dropout rates 
and increase on-track rates. An interaction between a 
supportive context and high-quality instruction might 
have both of these desirable effects. Since we found 
no differences between instruction in CHSRI schools 
and instruction in non-CHSRI schools serving similar 
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populations of students, we do not have any indications 
of the relation between the quality of instruction and 
dropout or on-track rates. 

Unfortunately, our quantitative analysis does not 
enable us to pinpoint the factors that are constrain- 
ing instructional reform efforts. This issue has been 
a focus of our qualitative work, however, and we have 
identified some possible explanations. First, like oth- 
ers we have found that instructional leadership is key 
to meaningful change.2 It is an important support for 
mobilizing teacher communities to collectively and 
coherently improve their classroom practices and for 
helping them focus and coordinate their improvement 
efforts. Furthermore, what we have found to date in- 
dicates that district leaders, small school reform staff, 
principals, and teachers have often been distracted 
from pursuing sustained and systematic instructional 
reform efforts by pressures associated with the imple- 
mentation of new schools and the multiple demands 
placed on individuals working in them. In addition, 
we found that accountability structures that encour- 
age schools to articulate their academic goals in terms 
of test scores and basic academic skills often did not 
lead schools to identify and implement strategies for 
reaching such goals. When strategies were identified, 
they often consisted of packaged programs linked to 
test preparation.3 

Instructional Improvement: Unanswered Questions 

It is not clear why, despite the relatively desirable 
teacher context for reform that we found in CHSRI 
schools, we did not see better results on our measures 
of Facilitators for Instructional Improvement. One 
possibility, of course, is that the ties between Boxes 6 
and 7 are not that strong, or they are heavily depen- 
dent on other factors such as Boxes 4 and 5. It may be 
that whether teachers have access to new ideas, quality 
professional development, and experience program 
coherence, for example, depends more on the actions 
of the principal and of others outside the school (such 
as CHSRI, CPS, and the state) than on teachers’ 
context. Clearly, these results indicate that those who 
hope to foster instructional improvement must do more 
than create small schools of committed, empowered, 
and collegial teachers. Our continuing qualitative 



inquiry into instructional reform efforts will look more 
closely at these issues and will explore the role played 
by CHSRI staff (Box 4) and by the district (Box 5) in 
addressing them. 

Indeed, the breakdown in the theory of action raises 
a host of important questions. Chief among them is 
the question of what actors such as the district or those 
implementing the reform (CHSRI) should do or not 
do. Can schools reform from within or does this find- 
ing indicate that outside supports and pressures (from 
Boxes 4 and 5, for example) are needed to foster efforts 
at meaningful change? Or are there ways in which 
these outside pressures are the problem rather than 
the solution? For example, are there ways in which 
broader systemwide goals and supports for instruction 
(pressures to implement test preparation programs or 
to attend to other district or CHSRI initiatives) crowd 
out school-based reform efforts? It is quite possible that 
both scenarios have merit. And to some extent, is it a 
matter of time? Do the particular challenges associated 
with implementation of small schools limit the time 
and attention given to meaningful change in class- 
room practice? Questions such as these deserve careful 
attention as this reform effort continues. 

In response to this concern, district leaders in 
Chicago are now coordinating their efforts with the 
CHSRI staff to help schools focus more directly and 
productively on instructional reform efforts. We are 
currently studying the implementation and impact of 
these efforts to see if they enable teachers to build on 
the sense of teacher collegiality, commitment, and trust 
in these schools to foster instructional change. 

Small School Reform in Chicago 

Given both the newness of the reform and the small 
size of our samples, it is clearly too soon to make broad 
claims about the efficacy of small school reform in 
Chicago. These findings however, do provide some 
reasons for cautious optimism about the reform and 
also highlight some issues in need of attention. First, as 
noted above, we see indications that small schools may 
be able to make a meaningful difference in students’ 
dropout rates. Our data indicate that the three-year 
dropout rate of students attending small schools is 7 
percentage points lower than it is for similar students 
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attending comparable schools. From some perspec- 
tives, a difference of this magnitude would justify the 
intervention. 

Our interest in this finding is strengthened by its 
alignment with the small schools’ theory of action. 
Central to the initiative’s rationale is the belief that 
smaller schools will foster contexts for students charac- 
terized by more personal attention, support, trust, and 
commitment. Our analysis of surveys indicates that 
this is happening to a meaningful extent for juniors. 
Although, as noted above, more research is needed to 
better understand the relationship between these con- 
texts and desired outcomes (particularly outcomes re- 
lated to lower dropout rates), the fact that dropout rates 
appear to be lower is certainly encouraging. Indeed, at 
this point in time, we feel that the marginally signifi- 
cant (p = 0.056) difference we identified between the 
dropout rates of CHSRI juniors and similar students 
attending comparable schools warrants giving small 
school reform more time and support, to see if these 
benefits persist. 



At the same time, this study raises some important 
concerns about the small school strategy — particularly 
as it relates to the need for instructional improvement. 
We see evidence that smaller schools enabled the cre- 
ation of contexts for teachers (ones characterized by 
greater trust, commitment, and sense of influence, 
for example), but these contexts do not appear to 
be fostering more systematic efforts at instructional 
improvement, different instructional practices, or 
improved performance on standardized tests. This 
kind of context may be desirable, but it is clearly not 
sufficient. 

Small schools appear to be fostering more personal- 
ized and supportive contexts for teachers and students. 
As the initiative continues to develop we will see 
whether these changes persist and whether schools, 
in partnership with CHSRI staff and district leaders, 
can successfully build on these more personalized and 
supportive contexts in ways that improve high school 
graduation rates and academic performance. 



Endnotes 

1 It is important to say at the onset of this discussion that we are 
limited in our ability to examine the relationship between the various 
components of our theory of action. It would be better if we could 
assess whether schools with high scores in a given component of the 
theory of action (say Box 6) also had higher scores in a related com- 
ponent (say Box 7) . Unfortunately, the limited number of schools in 
our sample constrains our ability to undertake such analysis. What we 
can do is assess whether, overall, small schools showed differences in a 
given component of our theory of action when compared to non- 
CHSRI schools serving similar populations and determine whether 
such differences between CHSRI and non-CHSRI schools appeared to 
carry over to related components of the theory of action. As the study 
continues, we will look further at these issues in an effort to develop a 
deeper understanding of the overall theory of action. 

2 Newman, et al. (2001); Sebring and Bryk (2000); Stevens (2006). 

3 Sporte, Correa, Kahne and Easton (2003); Stevens and Kahne 
(2005); Stevens (2006). 
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Commentary 



Joseph McDonald 

New York University 

This careful, thoughtful study of school reform in 
Chicago has sparked for this reader three related ideas 
on which I hang the following commentary. The hrst 
has to do with the nature of the phenomenon under 
study. The second has to do with how the researchers 
studied it. And the third has to do with what additional 
study seems needed. 

What Is the Phenomenon under Study? 

It is possible that what we learned to call high school 
in the 20^'^ century may he changing hack to what we 
thought of as high schooling^roxx^oxxt most of the 19'^^ 
century. Schooling for youth then was organized within 
multiple institutional designs. There was no normative 
design. Although it may not he obvious to the reader 
at the level of analysis the authors provide here, the 16 
schools they studied are varied in their institutional 
designs. The 12 that “converted” from some version 
of the “big” or normative high school design did so 
under the religious sway — so to speak — of different 
design ideas. In New York, where I live, one can visit 
small high schools that follow the Urban Assembly 
design, the Knowledge is Power Program (KIPP) 
design, the Coalition of Essential Schools design, the 
Expeditionary Eearning design, the Cristo Rey de- 
sign, and many others. The same is true in Chicago. 
For example, the authors report that two of the four 
started-from-scratch high schools they studied were Big 
Picture schools. The Big Picture high school design is 
starkly different from the normative one: substituting 
advisories and projects for courses, and placing students 



in community internships for two full days a week. 

“Wow,” we might say, stepping back from this study. 
“This is not just about small or big. High schools are 
getting to be different in the way that churches are 
different, or restaurants are different.” In the same 
instant, we might remember that adolescents differ 
too — easily as much as Episcopalians and Seventh-Day 
Adventists do in general, or McDonald’s and Spiaggia. 
Providing all youth an intellectually powerful educa- 
tion suitable for a 2T^ century society and economy 
will likely require multiple designs. 

Of course, the different small high school designs 
now proliferating in Chicago, New York, and other 
places have some overlapping features like intimacy, 
the deliberate cultivation of communities of practice, 
and a relative absence of bureaucracy. Reformers and 
funders promoting small high schools in Chicago, New 
York, and elsewhere often emphasize these overlapping 
features. The emphasis sometimes suggests that they 
mean to invent a new and better normative design, 
wherein something called “small high school” replaces 
something called “comprehensive high school.” One of 
the things I like about this study is that while its hnd- 
ings, in my view, encourage continued experimentation 
in high schooling, they also discourage expectation 
that a new one-best-model may be at hand. Something 
is happening, the researchers seem to say, as the result 
of the experimentation in Chicago to date. This is 
discernible, for example, in what juniors in the small 
high schools studied there report about their educa- 
tion. On the other hand, it’s not exactly clear from 
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the data what is happening. For example, while these 
schools do tend to have communities of practice — that 
is, their faculties do not operate in entirely cellular 
fashion — the communities of practice do not tend to 
alter ordinary instructional hahits. In the words of a 
previously published report from the Consortium on 
Chicago School Research, they tend to he more sup- 
portive than developmental.! 

Why is developing new kinds of instructional 
practice rather than supporting ordinary practice so 
important in Chicago high schooling? Consider the 
researchers’ finding that the CHSRI freshman class 
came into high school with an average reading score 
equal to the national average for sixth-graders. This 
means that many of these freshmen had a reading speed 
incapable of tracking what is ordinarily considered to be 
high school content. Many had word attack skills too 
undeveloped to read even many words that they know 
from listening, as well as many words they may never 
have heard but that are crucial to high school content: 
for example, photosynthesis, congruent, causality. This 
is a big deal. To educate such students well, high schools 
have to be prepared to do two things that hardly any are 
prepared to do. They have to mount intensive literacy 
interventions, and they have to immerse their students 
in intensive intellectual apprenticeships in real-world 
contexts. Without the second, the first will feel to the 
students like prison. Without the first, the second will 
be bewildering. Yet, whether small or large, most high 
schools are not organized for such instructional trans- 
formation — in terms of their use of space and time, in 
terms of the teaching skills they have on staff, in terms 
of their conceptions of curriculum. Still, as this study 
indicates, at least the small high schools are organized 
so that the adults in the building could put their heads 
together and figure out what they really need to do. It 
is just that the size and structure of their schools do not 
by themselves compel them to do so. As the researchers 
put it, “Clearly these results indicate that those who 
hope to foster instructional improvement must do more 
than create small schools of committed, empowered, 
and collegial teachers.” This is very important for 
reformers at all levels to know and accept. 

It is important also for the reformers to take note of a 
related warning from the researchers, one that has to do 



with how communities of practice might come to take 
bold steps to revise instructional habits. Do they need 
outside supports and pressures? Maybe, the researchers 
suggest. But outside supports and pressures can also 
be problematic. “Are there ways,” the researchers ask, 
“in which broader systemwide goals and supports for 
instruction (pressures to implement test preparation 
programs or to attend to other district or CHSRI initia- 
tives) crowd out school-based reform efforts?” 

How Do the Researchers Study the 
Phenomenon? 

In studying the Chicago experiment in high school 
redesign, the researchers have adopted a theory-based 
research design. Instead of treating the initiative they 
are studying as a unitary intervention based on clear 
intentions and predicted outcomes, they treat it as 
a complex intervention whose intentions have to be 
constructed before its outcomes can be charted and 
measured. In a 1998 monograph, Donald Schon and 
I referred to this process as asking reformers what they 
mean to do, then ascertaining whether they actually 
do what they mean.2 We wrote that this is not merely 
the obligation of researchers, but of reformers too who 
have to work together with the researchers in order to 
accomplish it. A theory of action is never given, Argyris 
and Schon argued in their seminal work on theory- 
based evaluation.3 It must be pieced together through 
continuous and collaborative inquiry. And at any point, 
it is inherently unstable. The political character of an 
initiative like CHSRI, and the range of actors and 
levels of operation involved, practically ensure that 
what reformers mean to do, and also what they actu- 
ally do, will vary across the initiative. It is the collective 
responsibility of the reformers and the researchers to 
strive over time to make the reform coherent. Schon 
was fond of describing the researcher’s task in this re- 
gard as running downhill, trying to get ahead of the 
reformer, holding up snapshots and asking, “Is this 
still what you mean? Do you know that you’re actually 
doing something else? Do you mean to do something 
else? Does the change make sense to you?” 

A theory of action is just a hypothesis constructed 
in order to account for a highly dynamic phenomenon. 
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We can see the implicit dynamism of the CHSRI 
theory of action that the researchers helped construct 
hy engaging in a little close reading of just its verbs. 
So small school reform will enable creation of school 
communities. Students and teachers who have chosen 
to he part of the school will act to enhance the com- 
munity. What does enable mean in this context? Isn’t it 
likely to mean different things to different people, and 
to emhed perhaps even contradictory theories whose 
contradictions have to he unearthed and confronted 
in order for the initiative to stand any chance at all of 
improving students’ lives? And what about have chosen'^. 
What does this verb phrase mean in the context of a 
l4-year-old and his or her parents picking, say, a Big 
Picture school versus some other design? And we can, 
of course, ask the same thing about the nouns that we 
do about the verbs: What does community mean? 

One of the great benefits of this study is that it 
does not stand alone. It is part of a now long and 
famous series of studies of school reform in Chicago 
undertaken by the Consortium on Chicago School 
Research in collaboration with the Chicago Public 
Schools. That is, it enjoys the great advantage — both 
in terms of validity and in terms of usefulness — of 
serving not some hopelessly summative purpose, but 
an ongoing formative one. It is one of those snapshots 
taken running downhill, as Schon put it — meant to 
engage respectful dialogue, meant to provoke possible 
tweaking or even substantial rethinking, raising hard 
questions that aim to strengthen efforts rather than 
shut them down. Chicago is very lucky to have the 
Consortium. 

What Other Studies Are Needed? 

Among the encouraging findings of this study is that 
CHSRI students have lower absence rates on aver- 
age than similar students in other schools. But the 
discouraging counterpart finding is that “on average, 
CHRSI students missed a lot of school — just over four 
full weeks of classes during the year.” It’s a finding 



that begs some more study — more snapshots running 
downhill, more material for the city’s dialogue. Why 
are these students missing so much school? For some 
kids who find themselves in schools full of bullying 
or other varieties of inhumane treatment including 
mindlessness, being absent a lot or even dropping out 
is a rational act. But the kids involved in this study, 
who miss on average a whole month of school each 
year, also report on average significantly more posi- 
tive school experiences (than similar students in other 
schools). So why are they staying out so much? Is it 
about asthma, working, taking care of siblings, taking 
care of their own children, family problems, neigh- 
borhood problems, housing problems, immigration 
status, depression? What factors show up in in-depth 
interviews of such kids? Is there anything that schools 
can do about the factors? Do they even fall within the 
scope of what the schools notice and talk about within 
their communities of practice? 

And this is only one of many other kinds of possible 
studies that would go below the surface of the data 
the researchers are continuing to collect, and likely 
prove valuable to the reform conversation in Chicago 
as elsewhere. Maybe some of these other studies are 
already in progress. I would obviously like to know a 
lot more than what I have been able to infer concern- 
ing the different “curricular themes or philosophical 
approaches to teaching and learning” that guide these 
schools or mean to guide them. And, of course, every 
reader of this paper will want follow-up study of what 
the researchers report as “a sizable shortcoming of the 
reform effort” to date, namely that instructional re- 
form, instructional practice, and test scores all appear 
the same at the study schools as at other CPS schools 
serving comparable students. Will the snapshots run- 
ning downhill manage to make a difference in this over 
time? Tune into the continuing dialogue. 



Endnotes 

1 Stevens with Kahne (2006). 

2 Schon and McDonald (1998). 

3 Argyris and Schon (1996). 



Commentary 



39 



Henry May 

Consortium for Policy Research in Education, 

University of Pennsylvania 

Education researchers have an informal responsibility 
to he conservative in their interpretations of results 
from their own research. I feel that the authors of this 
report have been justifiably conservative. Nevertheless, 
I take the liberty of being much less conservative in the 
following commentary because I feel that these early 
results demonstrate great potential for the Chicago 
High School Redesign Initiative (CHSRI) to show 
more dramatic effects in the years to come. 

To summarize, the present study produced the 
following results. The CHSRI schools showed much 
higher levels of teacher influence (effect size = 0.83) 
and considerably higher levels of teacher trust, collec- 
tive responsibility, and innovation (effect sizes rang- 
ing from 0.37 to 0.52). There were few differences in 
teachers’ activities related to instructional improvement 
(e.g., quality of professional development, access to 
new ideas) and mixed results with regards to students’ 
perceptions of instruction, expectations, and support. 
Effects on student outcomes included lower absentee- 
ism for freshman (20 vs. 26 days) and a lower dropout 
rate for juniors in CHSRI schools (20 vs. 27 percent). 
There were no detectable effects on student achieve- 
ment scores. 

Although these results are mixed, other research 
suggests that the large effects on teachers’ work en- 
vironments and their influence on instructional and 
social decisions may be a very important flrst stage 
in the reform’s development. Consider research by 
Ingersoll in which he shows that greater levels of 
teacher control (especially control over social and 
disciplinary issues in the school) are associated with 
lower levels of teacher turnover and lower levels of 
conflict among teachers and between teachers, stu- 
dents, and principals.! This suggests that teachers 
in CHSRI schools may have lower turnover rates 
in the coming years as a result of a more desirable 
work environment. 

Although the Consortium study found very few ef- 
fects on instructional improvement efforts, one must 
keep in mind that nearly every school affected by the 



No Child Eeft Behind Act (NCEB) is working fever- 
ishly to improve. While it may have been better to 
find positive differences associated with CHSRI, it is 
important to recognize that CHSRI schools showed 
the same reported level of activity and effort as com- 
parison schools. Assuming these activities and efforts 
are worthwhile, teachers will be able to improve their 
instruction. 

So consider this possibility: teachers in CHSRI 
schools work just as hard to build their skills as effective 
educators, but while teachers in other schools move on 
to more desirable teaching positions after they gain a 
certain number of hours of professional development or 
earn a Masters degree, teachers in CHSRI schools stay 
at their school because they enjoy their job and have 
more influence over how their school functions. That 
sounds a lot like a recipe for sustained improvement. 

There is also the possibility of an interesting interac- 
tion between student dropout rates and achievement 
scores — the lower dropout rate for juniors in CHSRI 
schools may be masking positive effects of CHSRI on 
student performance. It is well known that the simplest 
and easiest way to improve a school’s standardized test 
scores is to avoid testing the worst-performing students 
in the school. This is essentially what happens when 
students drop out of school: the worst-performing kids 
leave, causing the school’s average scores to increase. 

The implications for this study are relatively 
straightforward. The comparison schools have an 
advantage because a greater proportion of their lower- 
performing students dropped out of school before the 
statewide achievement test was administered, so the 
estimates of effects on student achievement from this 
study are probably conservative. The authors of this 
report have included controls for prior achievement, 
and this is likely to have mitigated the problem to 
some extent. Unfortunately, this is one of the most 
complicated statistical problems to solve. The sidebar 
below presents some additional statistical methods that 
might be used. 

To the extent possible, future research on the effects 
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of CHSRI should attend to issues of teacher turnover 
and student dropout rates. For example, effects on 
teacher turnover rates could he estimated hy moni- 
toring the proportion of teachers who switch schools 
or leave teaching completely. This extension of the 
CHSRI study might even allow for comparisons in 
reasons for turnover hy surveying teachers who move 
or leave. Future analyses of CHSRI data might also ex- 
plore alternative ways for estimating effects on student 
achievement after controlling for differences in student 
dropout rates as described in the sidehar. 



Overall, this report is an excellent example of a the- 
ory-driven evaluation, and the importance of capturing 
each outcome identified in the theory is clear in these 
early results. While there may not he strong evidence 
that the CHSRI initiative is producing positive effects 
on student achievement, there is strong evidence to 
support the claim that the CHSRI schools are produc- 
ing positive change in areas that are likely to lead to 
sustained improvement in instruction and ultimately, 
improved student achievement. 



The problem of taking prior dropouts into account 
when estimating school achievement is analogous 
to having a large amount of non-ignorable missing 
data. 2 Including a fixed effect at the school level 
for dropout rate is probably the simplest approach 
to addressing the problem, though doing this may 
produce an over-adjustment. A more statistically 
defensible approach might be to use multiple imputa- 
tion or Bayesian estimation, although this is relatively 



complicated. 3 Still another option is to match each 
CHSRI student to a comparison student based upon 
eighth-grade characteristics. If either student in a pair 
drops out of school, then the pair is excluded from 
analyses. Regardless of the method used, it would be 
difficult to adequately control for the differences in 
dropout rates without having enough information to 
predict with high accuracy each student’s propensity 
to drop out of school. 



I ji 



Endnotes 

1 Ingersoll (1996, 2003). 

2 Rubin (1987). 

3 See Dunn, Kadane, and Garrow (2003) for an example. 
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Appendix A 






Description of the Sample 

Teacher and Student Surveys 

T eachers and principals in all 104 non-alternative high schools were 
invited to participate in the Consortium survey, as were students in 
grades 9 through 12. A common core of items formed part of the survey for 
students in grades 9 through 12. In addition, there were some items that were 
answered only by students in grades 9 and 10, some items that were answered 
only by students in grade 11, and a large number of items that were unique 
to the twelfth-grade survey. Approximately 4,150 teachers in 87 schools 
responded, as did approximately 35,600 students in grades 9 through 10, 
and 10,600 students in grade 11. 

In all cases, only schools with at least a 50 percent survey response 
rate were included in our sample. We also chose not to include eight CPS 
achievement academies. Achievement academies are separate schools within 
a larger high school for students who did not meet the promotion criteria to 
attend a high school and are too old to remain in elementary school. Since 
their population, curriculum, and supports are radically different than what 
first-time freshmen experience, they are not part of our sample. Our final 
analytic sample for survey measures included approximately 3,400 teachers at 
52 schools and 16,000 first-time freshmen at 71 schools. Our analysis of the 
student context for eleventh-graders included 9,200 students in 54 schools. 



TABLE A1 

Consortium on Chicago Schooi Research Overaii Survey Response Rates 





Number of 
Schools 
Surveyed 


Number of 
Schools 
Responded 


Number of 
Individuals 
Surveyed 


Number of 
Individuals 
Who 

Responded 


Response 

Rate 

Among 

Surveyed 

Schools 


Response 

Rate 

Among 

Responding 

Schools 


Teacher Survey 


103 


87 


7,635 


4,142 


54% 


59% 


Ninth- and Tenth-Grade 


101 


87 


60,615 


35,608 


59% 


64% 


Student Survey 














Eleventh-Grade 


84 


73 


20,372 


11,003 


54% 


62% 



Student Survey 



TABLE A2 

Anaiytic Sampie* 



Number of Schools with Number of Individuals in 

Response Rates of 50% or More Those Schools Who Responded 



CHSRI Non-CHSRI CHSRI Non-CHSRI 



Teacher Survey 


10 


42 


220 


3,237 


First-Time Ninth-Grade 


12 


59 


865 


15,064 


Student Survey 










Eleventh-Grade 


6** 


48 


244 


9,016 


Student Survey 











* Number of individuals in the sample varied slightly depending on the measure being analyzed. See Appendix D. 
** A total of nine CHSRI schools served eleventh-graders. 



Student Outcomes 

We analyzed outcome data from all CPS high schools except alternative schools 
and the achievement academies described above. 



TABLE A3 

Students and Schools Used in Analyses of Cohort Dropout Rates 





Number of Students 


Number of Schools 




CHSRI 


Non-CHSRI 


CHSRI 


Non-CHSRI 


2002-03 Cohort 


447 


23,189 


5 


74 


2003-04 Cohort 


1,081 


25,013 


11 


75 


2004-05 Cohort 


1,615 


24,354 


16 


73 
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TABLE A4 

Students and Schools Used in the Analyses of Number of Days Absent 





Number of Students 


Number of Schoois 




CHSRi 


Non-CHSRi 


CHSRi 


Non-CHSRi 


First-Time Freshmen 
2002-03 


414 


21,596 


5 


68 


2003-04 


957 


23,037 


9 


67 


2004-05 


1,427 


22,208 


14 


63 


Juniors 

2002-03 


202 


17,311 


3 


67 


2003-04 


271 


17,140 


5 


67 


2004-05 


441 


16,717 


9 


70 


Note: Charter schools and the two Big Picture schools are not included for lack of data. 




TABLE AS 










Students and Schools Used in the Analysis of On-Track to Graduate Rates 






Number of Students 


Number of Schoois 




CHSRi 


Non-CHSRi 


CHSRi 


Non-CHSRi 


2002-03 


422 


21,960 


5 


68 


2003-04 


980 


23,583 


9 


67 


2004-05 


1,482 


22,888 


14 


64 


Note: Charter schools and the two Big Picture schools are not included for lack of data. 




TABLE AS 










Students and Schools Used in the Analysis of 

Prairie State Achievement Exam Scores in Reading (Math) 








Number of Students 


Number of Schoois 




CHSRi 


Non-CHSRi 


CHSRi 


Non-CHSRi 


2002-03 


200 (200) 


15,807 (15,509) 


3 


72 


2003-04 


286 (286) 


15,896(15,448) 


5 


72 


2004-05 


459 (459) 


17,139 (17.010) 


9 


73 
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Appendix B 



Rasch Analysis 

O nce the survey data were collected, we produced measures from 
multiple items on the survey using Rasch analysis^ These measures are 
more comprehensive and reliable than individual items. The Rasch approach 
permits the creation of a latent variable such as Collective Responsibility or 
Academic Press that is conceptually and empirically cohesive. Using items that 
relate to the same characteristic, a scale was constructed reflecting the relative 
“difficulty” (the likelihood that respondents will agree with a given item) 
of each item. 

The decision to omit or to include an item in the measure was based 
on the fit statistic, which has an expected value of 1 and is calculated by 
taking the mean squared deviations between the expected and observed 
values for that item. Items for which the fit statistic was greater than 1.3 were 
excluded. The scales were also evaluated using the person reliability statistic (the 
ratio of adjusted standard deviation to the root mean square error computed 
over the persons) which is approximately equivalent to Cronbach’s alpha. 

After the measures were created, the scores were converted to a 
10-point scale for ease of comparison within measures. However, since each 
measure represents a different latent characteristic, it is inappropriate to com- 
pare a score across measures — if a school has a mean score of 5.75 on Collective 
Responsibility and a mean score of 6.25 on Teacher Influence, it does not neces- 
sarily mean that the school is “better” on Influence than it is on Responsibility. 

Endnote 

1 Wright (1977). 
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Appendix C 






Description of Hierarchical Linear 
Models for Teacher and Student Survey 
Measures and Student Outcomes 

Teacher Measures 

T o examine whether teachers at CHSRI schools responded differently to 
survey measures than similar teachers at similar non-CHSRI schools, we 
used a three-level hierarchical linear model (HLM). At level one we adjusted for 
measurement error, which is produced by the Rasch analysis. At level two we mod- 
eled the teachers’ “true scores” by including teacher background characteristics 
that have been found to be related to measures of teacher context. We included 
gender, race, years of experience, level of education, and whether a person entered 
the profession through an alternative certification process. Since the survey was 
also taken by faculty members without specific classroom responsibility, we also 
controlled for whether a respondent was a classroom teacher. Slopes for these 
variables were fixed at level three, meaning that the relationship between each 
variable and the outcome measure was assumed to be the same across schools. 
The intercept in the level two equation can be interpreted as the school mean 
score on the measure after adjusting for teacher background characteristics. 

At level three we adjusted for characteristics of schools. We in- 
cluded a dummy variable indicating whether the school was a CHSRI 
school, and then used additional variables to control for background dif- 
ferences between CHSRI schools and other high schools. These variables 
included incoming eighth-grade achievement level of the student body, 
adjusted for current grade level, aggregate student socioeconomic sta- 
tus, and whether the school was at least 70 percent African American. 
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Level 1 

Measure 



jk 



=n 



^jk 



jk 



+ ^jk 



^jk 



where e ~ A^(0,1) > 5 '^.^ is the standard error estimated from the Rasch analysis 
for teacher y in school k and n is the teacher’s “true score.” 



Level 2 

= Pot + Pit (Female)^.^ + (Latino)^.^ + P 34 (African - American)^.^ + (3« (Asian)^.^ + 
Pjj. (Native American) + (3^^ (Other) (Classroom Teacher) + 

(Bachelor's Degree) (Doctorate Degree) + Pj„^ (3 or less years 
experience) + Pjjj, (4 to 10 years experience) + Pjj^ (Alternative Certification)^.^ + 
Put (No Gender Data)^^^ + Pj 4 j^ (No Race Data)^^^ + Pj;^ (No Data on Experience) + 
Pi 6 t (No Data on Education Degree) + Pj^^^ (No Classroom Teacher Information)^^ + 
Pjgj^ (No Alternative Certification Information)^.^^ + 

A description of these variables can he found in Table Cl. 



Level 3 

Pot “Too +Yoi(CNSRI)j + YQ2(Mean 8* Grade Achievement)^ +y„3(Mean Social Status),^ + 
y Q4 (Mean Concentration Poverty)^ + Y 05 (^ or more African - American),^ + 

P;,t = Y;,oTorp = ltol8. 

A description of these variables can be found in Table C2. 



Student Measures 

To examine whether students at CHSRI schools responded differently to survey 
measures than similar students at similar non-CHSRI schools we also used a three- 
level HLM. As with the analysis of teacher measures, at level one we adjusted for 
measurement error. At level two we modeled the students’ “true score” by including 
student background characteristics that have been found to be related to differ- 
ences in student attitudes and educational outcomes. For both first-time freshmen 
and juniors we included indicators of gender, race, socioeconomic status, and 
prior academic achievement as measured on the eighth-grade Iowa Tests of Basic 
Skills (ITBS) reading test, whether a student was receiving special education ser- 
vices, and whether a student was old for grade. For first-time freshmen we also 
included variables related to students’ mobility in elementary school and whether 
they attended a CPS school immediately prior to becoming a ninth grader. For 
juniors we included a set of dummy variables indicating how long students had 
been attending their current school. We again held the slopes of all these variables 
constant across schools. 

We adjusted for the same set of school characteristics at level three in the student 
measure analyses as we did in the analysis of teacher measures. Those variables 
included an indicator of whether the school was a CHSRI school, the average 
incoming achievement level of all current students, its average student socioeconomic 
status, and whether it was at least 70 percent African-American. 
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Level 1 

Measure 1 



where ~ A^( 0 , 1 ) , s is the standard error estimated from the Rasch analysis 
for student y in school k and 71 is the student’s “true score.” 

Level 2 

a. First-Time Freshmen: 

T^jk = Pot + Pit (Female)^., + pj, (Latino)^., + P3, (White)^., + P4, (Asian)^., + 

Pjj (Native American) (Social Status)^^ + P,^ (Concentration of Poverty) + 
Pgj^ (Special Education)^.^ + P,^ (8* Grade Achievement Reading)^^ + 

Pjq^ (Months Old Began Fligh School) + Pjj^ (From Other Public School) + 

Pi2t (From Private Elementary School)^^^ + Pjjj^ (Returning to CPS)^^ + Pj4^ (Moved 
1 time during Elementary School)^^ + Pj^^ (Moved 2 times during Elementary 
School)^^ + Pjgj (Moved 3 + times during Elementary School) + 

Pi7t (No Economic Data)^.j + P,gj (No 8* Grade Score)^.j^ + 

b. Juniors: 

9 

T^jk = Pot + X Ppt^^t + Plot (Old for Grade)^^ + P„^ (Been at school for less than 1 

p=\ 

year)^.j + ^^^k (Been at school for 2 years) + P^^ (Been at school for 3 years)^^ + 

P,4j^ (Been at school for 3 + years) + Pjjj. (No Economic Data)^^ + 

Pi6t (No 8* Grade Score)^.^ + r^.^, 

9 

where XPptA^yt represents 9 of the variables and coefficients common to the 
analyses of both first-time freshmen and juniors. 

A description of these variables can be found in Table Cl. 

Level 3 

Pot “Too +Yoi(f-FISRI)j^ -I- 7(12 (Mean 8* Grade Achievement),^ -l-7(,3(Mean Social Status),^ -l- 
y,,4(Mean Concentration Poverty),^ -1-705(70% or more African - American),^ -I- u^. 

P/<t - 'I pO’ Idr the rest of the variables. 

A description of these variables can be found in Table C 2 . 

Student Outcomes 

Dropout rates and freshman outcomes 

To examine whether student outcomes at CHSRI schools are different from student 
outcomes at non-CHSRI schools, we used a two-level hierarchical model adjusting 
for individual students’ characteristics and school-level characteristics. Level one 
represents students and level two represents schools. 
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At level one we controlled for previous performance in eighth grade, the social 
status in the census block where the student lives, the concentration of poverty in 
their block, gender, race, whether the student is receiving special education services, 
and whether the student is young or old at the beginning of high school. We also 
included variables related to students’ mobility in elementary school and whether 
they attended a CPS school immediately prior to becoming a ninth-grader. These 
slopes do not vary randomly at the school level. At level one there are also three 
dummy variables indicating the year the data represent. These three year variables 
are a function of school characteristics for that year and they are allowed to vary 
randomly at level 2. The school characteristics included an indicator of whether the 
school was a CHSRI school, the average incoming achievement level of all students 
in appropriate grades, the achievement level squared, and the school’s average student 
socioeconomic status (we tried some other school characteristics in the model that 
turned out not to have any explanatory power for the student outcomes). 

There is no intercept in the model. Therefore each of the three year variables 
represents the mean for that year for the variable analyzed, adjusted for students’ 
and schools’ characteristics. Since we centered all the variables on CHSRI students 
and CHSRI schools, the means for each year are the means for schools similar to 
CHSRI schools with similar students. 

The models and the variables in each level are described as follows: 

Level 1 

= Pij (8^^ Grade Achievement Reading),^. + (8* Grade Achievement Reading ^), + 

(Social Status ).j + (Concentration of Poverty),^. + P,^. (Male),^ + P^^. ( White ), + 

Py^. (Asian), -H Pg^. (Native American),^. + P,^. (Latino),^ -t Pj„^. (Special Education),^. + 

Piiy (No Economic Data),^. -H Pj 2 ^ (NoS"* Grade Score),^. + Pj,^. (Young Began High School),^, -t 
Pj 42 (Months Old Began High School),^. + Pjj^. (SlightlyOldBeganHigh School),^. + 

Pi 62 (Movedl time during Elementary School),^, -i- P^y (Moved 2 times during Elementary School), ^ + 
Pjg^. (Moved 3 + times during Elementary School),^ + Pj,^. (From Private Elementary School),^ + 

Pj„^. (Returning to CPS),^ + Pji^ (From Other Public Elementary School),^ + ^ 22 j (YEAR 02 - 03),^. + 
p23, (YEAR 03 - 04), + p^^, (YEAR 04 - 05), + r. 

Level 2 

P„ = Ypo forp = 1 through 21 

P 22 J -Y 220 +Y 221 (CHSRI 02 - 03), - 1-7222 (Mean 8* Grade Achievement 02 - 03 ) + 

7223 (Mean 8 * Grade Achievement 02 - 03^ ), - 1-7224 (Mean Social Status 02 - 03), + 

P 232 =Y 23 o +Y 231 (CHSRI 03 - 04), -H7232 (Mean 8* Grade Achievement 03 - 04), -r 

7,33 (Mean 8“’ Grade Achievement 03 - 04^), -H 7234 (Mean Social Status 03 - 04), -Hm 232 
P 242 =Y 24 o +Y 241 (CHSRI 04 - 05), - 1-7242 (Mean 8* Grade Achievement 04 - 05), + 

7243 (Mean 8 * Grade Achievement 04 - 05^), - 1-7244 (Mean Social Status 04 - 05), -Hm 242 



54 



Small High Schools on a Larger Scale 



For dropout rates and on-track to graduate indicator, 



/ 

};|(p. ~5(l,(p. ) andri. =log 

V 









/ 



For the number of days the student is absent during the year, 
-P{\Xj) andri^. =log^^) 



Eleventh- Grade Outcomes 

To examine whether students’ absences and test scores in eleventh grade at CHSRI 
schools are different from students’ absences and test scores at non-CHSRI schools, 
we used a two-level HLM, adjusting for individual students’ characteristics and 
school-level characteristics. As before, level one represents students and level two 
represents schools. 

Level 1 

h, = Pij Grade Achievement Reading ).j + (8* Grade Achievement Reading^), ^ -l- 

(Social Status )y + (Concentration of Poverty),^. -I- (Male),.. -I- P^^. (White),^. -l- 

(Asian),.. -I- P^^. (Native American) -I- P,^. (Latino),, -I- Pi,,, (Special Education),, + 

Pii, (No Economic Data),, -I- P12, (NoS* Grade Score),, -I- P13, (Old for Grade),, -l- 
P,„ (YEAR 02 - 03), + p,^, (YEAR 03 - 04), + p,„ (YEAR 04 - 05), + r,. 

Level 2 

^PJ =T;,o forp=l through 13 

Pmj =Yi 4 o +Yi 4 i (CHSRI 02 - 03), -I-Y 142 (Mean 8* Grade Achievement 02 - 03), -l- 

Yi43 (Mean 8“' Grade Achievement 02 - 03^ ), -I- Y144 (Mean Social Status 02 - 03), -I- u^^J 
Pi5 ; =Yi 5 o +Yi 5 i (CHSRI 03 - 04), -i-Yi 52 (Mcun S'*" Grade Achievement 03 - 04), -l- 

Yi53 (Means* Grade Achievement 03-04^), -I-Y154 (Mean Social Status 03 - 04), -I- Mjj, 
P16; =Yi 6 o +Y161 (CHSRI 04 - 05), -I-Y162 (Mean 8* Grade Achievement 04 - 05), -l- 

Y,63 (Mean 8* Grade Achievement 04 - 05^ ), -l- Yi64 (Mean Social Status 04 - 05), -I- Mj,, 

For the number of days the student is absent during the year, 
andri, =log(\.) 

and for the analysis of test scores, r],, = PS AE Score, . 
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Description of the Variables Used in the Models 

TABLE Cl 

Individual-Level Variables Used in Analyses 



WHERE USED DESCRIPTION 



Teacher Survey 


Gender was indicated by a dummy variable. 


Measure Analysis 


Race/Ethnicity was indicated by a set of dummy variables. Categories included “white, non- 
Hispanic,” “Hispanic,” “American Indian,” “Asian,” and “other” (which combined “biracial/ 
multi-ethnic” and “other”). 




Classroom Teachers were indicated by a dummy variable, distinguishing teachers who did not have 
a specific classroom responsibility from those who did. 




Education Level was indicated by a set of dummy variables. Bachelor’s Degree includes those who 
said their highest level of formal education was a bachelor’s degree, while Doctorate Degree 
includes those who indicated their highest level of formal education was a doctorate. Those with a 
Master’s degree or a Master’s degree plus 15 credits or more were combined in a single category. 




Teaching Experience was indicated by a set of dummy variables. The omitted category was teachers 
who indicated they had 1 1 to 15 or more than 15 years teaching experience. 




Entering Teaching Through an Alternative Certification Program was indicated by a dummy variable, 
distinguishing such teachers from those who entered the profession through other programs. 




Missing Background Values were imputed at the sample mean and a dummy variable was assigned. 


All Analyses of Students, 
Surveys, and Outcomes 


Gender was indicated by a dummy variable. 


Race/Ethnicity was indicated by a set of dummy variables including “white, non-Hispanic,” 
“Hispanic,” “African-American,” “American Indian,” and “Asian.” 




Socioeconomic Status included two variables. Social Status and Concentration of Poverty. Both 
were based on data from the 2000 U.S. Census information on the census block group in which 
students lived. Students’ home addresses were used to link each student to a particular block 
group within the city, which could then be linked to census data on the economic conditions 
of the student’s neighborhood. Four indicators were used to construct these variables: 1) log of 
the percentage of families above the poverty line, 2) log of the percentage of men employed in 
the block group, 3) the average level of education among adults over age 21, and 4) log of the 
percentage of men in the block group employed as managers or executives. The first two of these 
were reverse coded and combined into the variable called Concentration of Poverty, while the 
other two indicators were combined into the variable called Social Status. The census data allow 
for a more accurate indicator of students’ economic status than a simple indicator of whether the 
student qualifies for free or reduced lunch. The vast majority of students in CPS qualify for free 
or reduced-cost lunch, and there is wide variation in the economic status of students who qualify 
as low income. Furthermore, by the time students reach high school age, proportionately more 
parents fail to apply for free and reduced lunch, and different schools treat this phenomenon 
differently. 
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DESCRIPTION 



WHERE USED 



Receiving Special Education Services based on administrative records. 


All Analyses of Students, 


Students’ Elementary Achievement was constructed based on student reading scores on the Iowa 
Tests of Basic Skills (ITBS) they took in eighth grade. The analyses of student outcomes included 
linear and quadratic terms for this variable. 


Surveys, and Outcomes 


Missing Values. If students were missing values on the socioeconomic variables or on their eighth- 
grade achievement, they were given values at the sample mean and assigned a dummy variable. 




Age on Entry into High School was distinguished for first-time ninth graders by a set of variables. 
Months Old Began High School gave the number of months older than 14 years, 8 months that 
a student was as of September 1 of the academic year. Students older than 14 years, 8 months 
should have started school with the previous cohort if they were to follow school-system guidelines. 
Slightly Old Began High School is a dummy variable that indicates students between 14 years 
and 9 to 11 months old when they began high school. “Young Began High School” is a dummy 
variable indicating whether a student was younger than age 14 at the start of high school. 


Additional Variables: 
Analyses of First-Time 
Freshmen Only 


School Mobility in Elementary School was included in the models as a proxy of educational stability. 
Three dummy variables were constructed identifying students who moved once, twice, or three 
or more times in the three years prior to entering high school. 




New Students (those who did not enter their CPS high school from a CPS elementary school) 
make up about 10 percent of each cohort and present measurement difficulties. For these 
students, we do not have information on elementary mobility or incoming achievement. Three 
dummy variables were used to identify these students, since their attitudes could presumably be 
influenced by their prior school experience. From Other Public School includes those students 
who had never before attended a CPS school and entered high school from another public school. 
From Private Elemntary School includes those who entered high school from a private elementary 
or middle school, and Returning to CPS includes students returning to CPS after a time away. 
Sample-wide mean achievement and mobility were then imputed for these students. 




Whether a Student Was Old for Grade was indicated by a dummy variable to show whether a student 
was older than what would be expected based on school system guidelines. 


Additional Variables: 
Analyses of Junior 
Students Only 


Length of Time in Current High School was hypothesized to be a characteristic that could play an 
important role in student attitudes toward their school. We used a system of three dummy variables 
to capture this. Students who had been at the school for a year was the omitted category. 




Academic Year was captured by a set of three dummy variables representing the three years of 
this study. 


Additional Variables: 
Student Outcome 
Analyses Only 
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TABLE C2 

School-Level Variables Used in Analyses 


WHERE USED 


DESCRIPTION 


All Analyses 


Whether a School is a CHSRI School was indicated with a dummy variable. 




School-Level Achievement. For the analyses of survey measures this variable was computed by 
averaging the eighth-grade ITBS reading score of all current members of a school’s student body, 
adjusting for current grade level. For the analyses of student outcomes this variable was computed 
by averaging the eighth-grade ITBS reading score of first-time freshmen or eleventh- grade 
students at the school depending on the outcome analyzed. The analyses of student outcomes 
included linear and quadratic terms for this variable. 




School-Level Socioeconomic Status. A description of how this variable was created at the individual 
level can be found in Table Cl. The school-level indicator was an average of the social status of all 
students in the school based on the census block in which each lived. 


Additional Variables: 
Survey Analyses Only 


School-Level Concentration of Poverty. A description ofhow this variable was created at the individual 
level can be found in Table Cl. The school-level indicator was an average of the concentration of 
poverty of all students in the school based on the census block in which each lived. 




Racial Composition of the School. A dummy variable was included for schools whose student body 
was at least 70 percent African-American. 
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Appendix D 



Description of Teacher and Student Survey Measures 

TABLE D1 

Teacher Survey Measure Descriptions and Statistics 



Measure 


Description 


N 


Individual 

Reliability 


Between 

School 

Reliability 


Variation 

Between 

Schools 


Access to 
New Ideas 


The extent to which teachers participate in professional 
development. Questions ask teachers how often they attend 
professional development activities sponsored by the school, 
district, or union; take continuing education courses at a college 
or university; and network with teachers from other schools. 


3,140 


0.67 


0.46 


4.7% 


Collective 

Responsibility 


Teachers’ assessment of the strength of their shared commitment 
to improve the school so that all students learn. Questions ask 
teachers how many colleagues feel responsible for students’ 
academic and social development, set high standards for 
professional practice, and take responsibility for school 
improvement. 


3,225 


0.92 


0.79 


9.7% 


Commitment 
to Innovation 


Teachers’ perceptions of whether they are continually learning 
and seeking new ideas, have a “can do” attitude, and are 
encouraged to try new ideas in their teaching. 


3,230 


0.89 


0.83 


13.2% 


Expectations 
for Post- 
Secondary 
Education 


Teachers’ reports of the degree to which they expect that most 
students at their school will go to college. Questions ask if 
teachers feel it is part of their job to prepare students to succeed 
in college and if they spend time out of class in helping students 
plan for college. 


3,067 


0.82 


0.93 


35.5% 


Principal 

Instructional 

Leadership 


Teachers’ perception of their principal as an instructional 
leader with respect to the teaching and learning standards, 
communication of a clear vision for the school, and tracking 
of academic progress. 


3,194 


0.86 


0.87 


16.5% 


Program 

Coherence 


The degree to which teachers feel the programs at their school 
are coordinated with each other and with the school’s mission. 
Questions ask teachers if instructional materials are consistent 
within and across grades and if there is sustained attention to 
quality program implementation. 


3,181 


0.74 


0.78 


11.2% 
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TABLE D1-C0NTINUED 

Teacher Survey Measure Descriptions and Statistics 



Measure 


Description 


N 


Individual 

Reliability 


Between 

School 

Reliability 


Variation 

Between 

Schools 


Quality 

Professional 

Development 


Teachers’ assessment of the degree to which professional 
development has influenced their teaching, helped them 
understand students better, and provided them with 
opportunities to work with colleagues as well as teachers 
from other schools 


3,135 


0.81 


0.75 


9.3% 


Quality Student 
Discussion 


Teachers’ reports of how well students interact with each other in 
classroom discussions. Questions ask whether students build off 
each others’ ideas, show respect, provide constructive feedback, 
and use data and text references to support their ideas. 


2,397 


0.72 


0.62 


11.3% 


Reflective 

Dialogue 


Teachers’ assessment of how often they talk with one another 
about instruction and student learning. Questions ask teachers 
about their discussion of curriculum and instruction, the school’s 
goals, and the best ways to help students learn and manage 
classroom behavior. 


3,253 


0.77 


0.71 


7.9% 


Teacher 

Influence 


Measures the extent of teachers’ involvement in school decision 
making. It assesses teachers’ influence on the selection of 
instructional materials, setting of school policy, in-service 
program planning, spending of discretionary funds, and hiring 
of professional staff 


3,258 


0.84 


0.89 


21.9% 


Teacher- 

Teacher 

Trust 


The extent to which teachers feel they have mutual respect 
for each other, for those who lead school improvement efforts, 
and for those who are experts at their craft. Questions also ask 
teachers if they feel comfortable discussing their feelings and 
worries and really care about each other. 


3,245 


0.82 


0.79 


11.2% 
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TABLE D2 



Student Measure Descriptions and Statistics for First-Time Freshmen (Juniors) 



Measure 


Description 


N 


Individual 

Reliability 


Between 

School 

Reliability 


Variation 

Between 

Schools 


Academic 

Press 


Students’ views of their teachers’ efforts to push students to 
higher levels of academic performance. Students also report on 
the degree to which they find their classes to he challenging. 


12,481 

(7,227) 


0.76 


0.75 

(0.72) 


5.0% 

(5.6%) 


Classroom 

Personalism 


The degree to which students perceive that their teachers give 
individual attention to and are concerned about their students’ 
academic performance 


12,376 

(7,192) 


0.74 


0.80 

(0.83) 


5.5% 

(9.2%) 


Quality 

English 

Instruction 


Student’s reports of the frequency with which they are made 
to practice English activities such as writing papers of three 
or more pages, rewriting in response to comments, discussing 
connections between a reading and real life people, and 
understanding how authors are influenced and the tools they 
use to convey meaning. 


14,294 

(8,443) 


0.86 


0.84 

(0.83) 


5.4% 

(8.6%) 


Quality 

Math 

Instruction 


Student’s reports of the frequency that they are made to 
practice math activities such as writing problems for other 
students to solve, applying math to situations outside the 
classroom, and explaining to the class how they solved a 
problem. 


14,590 

(8,541) 


0.76 


0.88 

(0.87) 


8.9% 

(10.0%) 


Peer Support 
for Academic 
Achievement 


The norms among students with regard to their peers’ support 
of academic work. Questions ask if students talk about what 
they did in class, if they help each other with homework and 
test preparation, and if their friends think it is important to 
attend class. 


15,592 

(9,323) 


0.84 


0.87 

(0.84) 


6.3% 

(7.2%) 


Respectful 

Classroom 

Behavior 


Students’ assessment of their peers’ classroom behavior with 
regard to how they treat each other, how often they disrupt 
class, if they have respect for each other, and if they help each 
other learn. 


12,297 

(7,143) 


0.74 


0.80 

(0.85) 


61.3% 

(42.1%) 


Schoolwide 

Future 

Qrientation 


Student’s reports of the degree to which teachers work hard to 
make sure that all students are learning, are staying in school, 
are planning for their futures; and the degree to which all 
students are encouraged to go to college. 


15,694 

(9,395) 


0.83 


0.95 

(0.91) 


17.2% 

(15.5%) 


Sense of 
Belonging 


Students’ reports of how personally connected they feel to the 
school. Students rate the degree to which the people at school 
feel like family, whether people at school care if they come to 
school, and whether they participate in activities at the school. 


15,619 

(9,365) 


0.81 


0.87 

(0.84) 


7.6% 

(9.1%) 


Student- 
Teacher Trust 


Students’ perceptions about the quality of their relationships 
with teachers. Questions ask students if teachers care about 
them, keep promises, listen to their ideas, and try to be fair. 


15,829 

(9,501) 


0.78 


0.92 

16.5% 


16.5% 

(16.7%) 


Teacher 

Support 


Students’ reports of teachers’ being there to help with personal 
matters. Students were asked whether there is a teacher who they 
can talk to about personal problems, who gives extra help with 
schoolwork, and who cares about how the student is doing. 


15,769 

(9,459) 


0.80 


0.92 

(0.87) 


12.7% 

(11.0%) 
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